[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 985389986 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003293] x2apic enabled [ 0.005005] Switched APIC routing to physical x2apic. [ 0.006018] kvm-guest: setup PV IPIs [ 0.009407] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011013] pid_max: default: 32768 minimum: 301 [ 0.013091] LSM: Security Framework initializing [ 0.014054] Yama: becoming mindful. [ 0.015047] SELinux: Initializing. [ 0.017046] *** VALIDATE selinux *** [ 0.030000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.037416] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.039076] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.040113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.041116] *** VALIDATE tmpfs *** [ 0.044259] *** VALIDATE proc *** [ 0.045284] *** VALIDATE cgroup *** [ 0.046015] *** VALIDATE cgroup2 *** [ 0.047327] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.049159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.050012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.051033] Spectre V2 : User space: Vulnerable [ 0.052014] Speculative Store Bypass: Vulnerable [ 0.058288] debug: unmapping init [mem 0xffffffffa7859000-0xffffffffa7860fff] [ 0.060000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.060668] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.061028] ... version: 2 [ 0.062020] ... bit width: 48 [ 0.063017] ... generic registers: 4 [ 0.064022] ... value mask: 0000ffffffffffff [ 0.065022] ... max period: 00007fffffffffff [ 0.066019] ... fixed-purpose events: 3 [ 0.067025] ... event mask: 000000070000000f [ 0.069165] rcu: Hierarchical SRCU implementation. [ 0.073415] smp: Bringing up secondary CPUs ... [ 0.074601] x86: Booting SMP configuration: [ 0.075414] .... node #0, CPUs: #1 #2 #3 [ 0.093022] smp: Brought up 1 node, 4 CPUs [ 0.095014] smpboot: Max logical packages: 1 [ 0.096014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.173450] node 0 deferred pages initialised in 73ms [ 0.177362] devtmpfs: initialized [ 0.180597] x86/mm: Memory block size: 128MB [ 0.187034] gcov: version magic: 0x41383552 [ 0.190547] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.191078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.192264] pinctrl core: initialized pinctrl subsystem [ 0.194221] [ 0.195009] ************************************************************* [ 0.196016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197016] ** ** [ 0.198017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.199016] ** ** [ 0.200015] ** This means that this kernel is built to expose internal ** [ 0.201018] ** IOMMU data structures, which may compromise security on ** [ 0.202020] ** your system. ** [ 0.203016] ** ** [ 0.204013] ** If you see this message and you are not debugging the ** [ 0.205014] ** kernel, report this immediately to your vendor! ** [ 0.206015] ** ** [ 0.207019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208018] ************************************************************* [ 0.210605] NET: Registered protocol family 16 [ 0.211722] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.212074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.216052] cpuidle: using governor menu [ 0.220318] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.225661] PCI: Using configuration type 1 for base access [ 0.228141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.240982] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.241099] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.243033] cryptd: max_cpu_qlen set to 1000 [ 0.245267] ACPI: Added _OSI(Module Device) [ 0.246000] ACPI: Added _OSI(Processor Device) [ 0.247016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.261621] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.280056] ACPI: Interpreter enabled [ 0.281135] ACPI: PM: (supports S0 S3 S4 S5) [ 0.282017] ACPI: Using IOAPIC for interrupt routing [ 0.283124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.284989] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.304590] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.305470] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.306021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.307078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.310200] acpiphp: Slot [2] registered [ 0.312090] acpiphp: Slot [5] registered [ 0.313190] acpiphp: Slot [6] registered [ 0.314194] acpiphp: Slot [7] registered [ 0.315176] acpiphp: Slot [8] registered [ 0.316164] acpiphp: Slot [9] registered [ 0.317154] acpiphp: Slot [10] registered [ 0.318137] acpiphp: Slot [3] registered [ 0.319119] acpiphp: Slot [4] registered [ 0.321020] acpiphp: Slot [11] registered [ 0.322127] acpiphp: Slot [12] registered [ 0.323329] acpiphp: Slot [13] registered [ 0.324719] acpiphp: Slot [14] registered [ 0.325571] acpiphp: Slot [15] registered [ 0.327103] acpiphp: Slot [16] registered [ 0.328171] acpiphp: Slot [17] registered [ 0.329324] acpiphp: Slot [18] registered [ 0.330141] acpiphp: Slot [19] registered [ 0.331142] acpiphp: Slot [20] registered [ 0.332118] acpiphp: Slot [21] registered [ 0.333144] acpiphp: Slot [22] registered [ 0.334659] acpiphp: Slot [23] registered [ 0.335141] acpiphp: Slot [24] registered [ 0.336127] acpiphp: Slot [25] registered [ 0.337129] acpiphp: Slot [26] registered [ 0.338137] acpiphp: Slot [27] registered [ 0.339146] acpiphp: Slot [28] registered [ 0.340156] acpiphp: Slot [29] registered [ 0.341156] acpiphp: Slot [30] registered [ 0.342129] acpiphp: Slot [31] registered [ 0.343076] PCI host bridge to bus 0000:00 [ 0.344025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.345027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.346664] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.348023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.349029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350511] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.353201] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.356996] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.368017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.373060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.374021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378307] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380451] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.381054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384533] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.400015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.407788] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.410016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417044] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.425023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.432739] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436033] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.439026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.446027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.454200] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.457018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.461025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.469025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.475908] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.478023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.482000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.488023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.495428] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.499023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.502036] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.509030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.515471] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.518029] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.521031] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.528024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.536662] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.538400] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.539382] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.540664] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.541281] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.548036] iommu: Default domain type: Passthrough [ 0.549668] SCSI subsystem initialized [ 0.550381] ACPI: bus type USB registered [ 0.551213] usbcore: registered new interface driver usbfs [ 0.552121] usbcore: registered new interface driver hub [ 0.553108] usbcore: registered new device driver usb [ 0.554214] pps_core: LinuxPPS API ver. 1 registered [ 0.555017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.556169] PTP clock support registered [ 0.557604] EDAC MC: Ver: 3.0.0 [ 0.559216] PCI: Using ACPI for IRQ routing [ 0.561089] NetLabel: Initializing [ 0.562017] NetLabel: domain hash size = 128 [ 0.563014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.564112] NetLabel: unlabeled traffic allowed by default [ 0.566178] vgaarb: loaded [ 0.568442] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.570000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.574177] clocksource: Switched to clocksource kvm-clock [ 0.848920] VFS: Disk quotas dquot_6.6.0 [ 0.853889] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.859956] *** VALIDATE ramfs *** [ 0.862581] *** VALIDATE hugetlbfs *** [ 0.867797] pnp: PnP ACPI init [ 0.876795] pnp: PnP ACPI: found 6 devices [ 0.914185] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.919407] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.924752] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.931864] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.937413] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.942825] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.948319] NET: Registered protocol family 2 [ 0.953509] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.972798] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.981522] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.995725] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.008532] TCP: Hash tables configured (established 65536 bind 65536) [ 1.017458] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.035418] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.043558] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.051246] NET: Registered protocol family 1 [ 1.060034] RPC: Registered named UNIX socket transport module. [ 1.066842] RPC: Registered udp transport module. [ 1.076201] RPC: Registered tcp transport module. [ 1.083189] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.092179] NET: Registered protocol family 44 [ 1.099826] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.106162] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.114311] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.122260] PCI: CLS 0 bytes, default 64 [ 1.128900] Unpacking initramfs... [ 2.723222] hrtimer: interrupt took 5482412 ns [ 4.284978] debug: unmapping init [mem 0xffff973afcc54000-0xffff973afffbffff] [ 4.289804] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.292660] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.296244] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.535062] Initialise system trusted keyrings [ 7.558101] Key type blacklist registered [ 7.573720] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.649880] zbud: loaded [ 7.674141] *** VALIDATE nfs *** [ 7.675545] *** VALIDATE nfs4 *** [ 7.695825] pstore: using deflate compression [ 7.743906] Platform Keyring initialized [ 8.547660] NET: Registered protocol family 38 [ 8.549875] Key type asymmetric registered [ 8.554608] Asymmetric key parser 'x509' registered [ 8.557565] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 8.564835] io scheduler mq-deadline registered [ 8.567831] io scheduler kyber registered [ 8.571440] io scheduler bfq registered [ 8.575665] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 8.586257] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 8.593990] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 8.611183] ACPI: Power Button [PWRF] [ 8.623248] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.646625] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 8.704877] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 8.712594] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.734309] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.835631] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.935118] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.955905] Non-volatile memory driver v1.3 [ 8.961614] Linux agpgart interface v0.103 [ 9.184728] virtio_blk virtio1: [vda] 149840 512-byte logical blocks (76.7 MB/73.2 MiB) [ 9.204564] vda: detected capacity change from 0 to 76718080 [ 9.272377] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.282916] vdb: detected capacity change from 0 to 1073741824 [ 9.345943] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.349484] vdc: detected capacity change from 0 to 2621440000 [ 9.371956] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.374925] vdd: detected capacity change from 0 to 2621440000 [ 9.391892] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.417268] vde: detected capacity change from 0 to 4294967296 [ 9.467644] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.481443] vdf: detected capacity change from 0 to 4294967296 [ 9.531826] libphy: Fixed MDIO Bus: probed [ 9.617174] usbcore: registered new interface driver usbserial_generic [ 9.630326] usbserial: USB Serial support registered for generic [ 9.659762] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.665703] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.667910] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.672313] mousedev: PS/2 mouse device common for all mice [ 9.676252] rtc_cmos 00:05: RTC can wake from S4 [ 9.684077] rtc_cmos 00:05: registered as rtc0 [ 9.685953] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.688258] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.695056] intel_pstate: CPU model not supported [ 9.700550] hid: raw HID events driver (C) Jiri Kosina [ 9.703310] usbcore: registered new interface driver usbhid [ 9.705773] usbhid: USB HID core driver [ 9.708154] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.709106] drop_monitor: Initializing network drop monitor service [ 9.714992] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.716601] Initializing XFRM netlink socket [ 9.718952] NET: Registered protocol family 10 [ 9.723919] Segment Routing with IPv6 [ 9.735144] NET: Registered protocol family 17 [ 9.739409] mpls_gso: MPLS GSO support [ 9.747831] RAS: Correctable Errors collector initialized. [ 9.750419] AVX version of gcm_enc/dec engaged. [ 9.752562] AES CTR mode by8 optimization enabled [ 9.836953] sched_clock: Marking stable (9836915484, 0)->(12337420083, -2500504599) [ 9.843237] registered taskstats version 1 [ 9.845825] Loading compiled-in X.509 certificates [ 9.848831] zswap: loaded using pool lzo/zbud [ 9.887539] Key type big_key registered [ 9.913241] Key type encrypted registered [ 9.914723] ima: No TPM chip found, activating TPM-bypass! [ 9.916863] ima: Allocated hash algorithm: sha1 [ 9.919501] ima: No architecture policies found [ 9.920930] evm: Initialising EVM extended attributes: [ 9.922760] evm: security.selinux [ 9.923699] evm: security.ima [ 9.924528] evm: security.capability [ 9.925859] evm: HMAC attrs: 0x1 [ 9.931570] rtc_cmos 00:05: setting system clock to 2026-10-02 17:45:45 UTC (1790963145) [ 9.940398] debug: unmapping init [mem 0xffffffffa8803000-0xffffffffa89fffff] [ 9.944745] debug: unmapping init [mem 0xffffffffa7582000-0xffffffffa7858fff] [ 9.962367] Write protecting the kernel read-only data: 28672k [ 9.975571] debug: unmapping init [mem 0xffffffffa5c03000-0xffffffffa5dfffff] [ 9.983653] debug: unmapping init [mem 0xffffffffa6514000-0xffffffffa65fffff] [ 10.078535] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.087720] systemd[1]: Detected virtualization kvm. [ 10.089642] systemd[1]: Detected architecture x86-64. [ 10.091876] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.128528] systemd[1]: No hostname configured. [ 10.130385] systemd[1]: Set hostname to . [ 10.132826] random: systemd: uninitialized urandom read (16 bytes read) [ 10.138806] systemd[1]: Initializing machine ID from random generator. [ 10.239553] random: ln: uninitialized urandom read (6 bytes read) [ 10.328276] random: systemd: uninitialized urandom read (16 bytes read) [ 10.331775] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 10.340341] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 10.354593] systemd[1]: Starting Create Volatile Files and Directories... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.184463] device-mapper: uevent: version 1.0.3 [ 12.186656] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 14.605467] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 14.852567] random: fast init done [ 15.310587] scsi host0: ata_piix [ 15.512700] scsi host1: ata_piix [ 15.556742] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 15.584689] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 20.442888] random: crng init done [ 20.444410] random: 7 urandom warning(s) missed due to ratelimiting [ 24.045128] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 27.386282] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 35.138113] printk: systemd: 26 output lines suppressed due to ratelimiting [ 37.241649] SELinux: Disabled at runtime. [ 37.707591] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 37.778166] systemd[1]: Detected virtualization kvm. [ 37.807067] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 42.301055] systemd[1]: initrd-switch-root.service: Succeeded. [ 42.307041] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 42.330887] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 42.369678] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 42.385093] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 42.514882] systemd[1]: Starting Journal Service... Starting Journal Service... [ 42.566246] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 42.823172] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ 42.907121] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ 42.942197] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ 43.001161] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 43.003425] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 43.055055] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ 43.139023] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ 43.164325] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ 43.191639] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 43.231335] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 43.274722] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 46.892305] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 49.569428] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 49.579603] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 51.280270] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 51.418174] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (11s / no limit) [** ] A start job is running for Configur…only root support (12s / no limit) [*** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit)[ 56.416429] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (14s / no limit)[ 57.032485] NFS: Registering the id_resolver key type [ 57.035436] Key type id_resolver registered [ 57.039640] Key type id_legacy registered [ *] A start job is running for Configur…only root support (15s / no limit) [ **] A start job is running for Configur…only root support (15s / no limit) [ ***] A start job is running for Configur…only root support (16s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg411-server login: [ 134.000558] libcfs: loading out-of-tree module taints kernel. [ 134.052224] Key type ._llcrypt registered [ 134.060256] Key type .llcrypt registered [ 134.184193] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 154.084372] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 155.789463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 155.820860] alg: No test for adler32 (adler32-zlib) [ 157.554154] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 158.588026] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 160.424349] Key type lgssc registered [ 162.352082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 183.302249] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 225.660690] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 237.559299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 237.585119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.815245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 238.876407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 238.975824] Lustre: lustre-MDT0000: new disk, initializing [ 239.084261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.100444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 243.589828] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.237313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 257.354759] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 257.407804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 257.414857] Lustre: Skipped 1 previous similar message [ 257.517334] Lustre: lustre-MDT0001: new disk, initializing [ 257.604229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.644102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 257.667203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 262.551839] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.702058] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 278.384713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 278.796313] Lustre: lustre-OST0000: new disk, initializing [ 278.805078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 278.820753] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 278.950683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 282.033394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 282.047541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 282.153410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 286.363527] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 301.850244] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 301.984565] Lustre: lustre-OST0001: new disk, initializing [ 301.989638] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 301.995490] Lustre: 9527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 302.070627] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 308.531824] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 308.846974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 308.856318] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 308.935385] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 320.348508] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 331.223126] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 338.530182] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing check_logdir /tmp/testlogs/ [ 344.672472] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing yml_node [ 351.514639] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 355.372565] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 359.698628] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 362.609351] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 13:51:36 EDT 2026 [ 381.714442] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 383.028891] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 400.864854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 400.870429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 400.882054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 401.377816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.403312] Lustre: Skipped 1 previous similar message [ 401.416596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 401.421650] Lustre: Skipped 3 previous similar messages [ 404.946245] Lustre: server umount lustre-MDT0000 complete [ 408.534792] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963544 with bad export cookie 9916287082561426391 [ 408.538101] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 408.547124] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 411.617061] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.618727] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.641532] Lustre: Skipped 1 previous similar message [ 411.642926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 411.660478] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 413.666925] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 413.667510] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 413.686884] Lustre: Skipped 2 previous similar messages [ 414.853789] Lustre: server umount lustre-MDT0001 complete [ 418.788222] Lustre: server umount lustre-OST0000 complete [ 422.409227] Lustre: server umount lustre-OST0001 complete [ 428.913379] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 436.756736] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 468.508486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.712401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 468.747566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 468.790601] Lustre: lustre-MDT0000: new disk, initializing [ 468.879658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.912928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 472.392832] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 480.803152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.898421] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 480.941410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 480.944749] Lustre: Skipped 1 previous similar message [ 481.029346] Lustre: lustre-MDT0001: new disk, initializing [ 481.112842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 481.137448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 481.144579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 484.361440] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 490.777204] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 495.361136] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 502.313902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 502.719531] Lustre: lustre-OST0000: new disk, initializing [ 502.723460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 502.732955] Lustre: 17050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 502.831354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 504.653369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 504.666992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 504.858324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 513.833625] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 530.606051] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 541.197919] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 541.671534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 549.619430] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 550.037669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 556.001686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 556.006337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.012151] LustreError: Skipped 1 previous similar message [ 556.016099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 556.041853] Lustre: Skipped 2 previous similar messages [ 558.691819] Lustre: server umount lustre-OST0000 complete [ 566.243050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.247171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 566.257615] Lustre: Skipped 1 previous similar message [ 566.261421] Lustre: Skipped 1 previous similar message [ 569.924304] Lustre: server umount lustre-MDT0000 complete [ 571.361520] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.372756] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 573.223969] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963708 with bad export cookie 9916287082561427924 [ 573.233305] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.237239] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 573.543184] Lustre: server umount lustre-MDT0001 complete [ 576.930934] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 13:55:11 (1790963711) [ 584.655869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.162507] LustreError: 19317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.226251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.599281] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 590.305872] LustreError: 19318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.692995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.988902] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 608.297890] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 613.271834] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 618.641315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 623.556806] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 629.922919] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 635.367833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 635.368713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.383462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 635.393526] Lustre: Skipped 1 previous similar message [ 635.407720] Lustre: Skipped 2 previous similar messages [ 637.257093] Lustre: server umount lustre-OST0000 complete [ 646.633271] Lustre: server umount lustre-MDT0000 complete [ 650.726472] LustreError: 22007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.771753] LustreError: 22007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 652.046211] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963787 with bad export cookie 9916287082561428806 [ 652.067636] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.244993] Lustre: server umount lustre-MDT0001 complete [ 673.663211] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 679.685076] Key type lgssc unregistered [ 680.293916] LNet: 22869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.323257] LNetError: 22869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 680.404158] LNet: Removed LNI 192.168.204.111@tcp [ 682.666164] Key type .llcrypt unregistered [ 682.674200] Key type ._llcrypt unregistered [ 711.550849] Key type ._llcrypt registered [ 711.554190] Key type .llcrypt registered [ 711.704282] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 713.415056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 713.563603] alg: No test for adler32 (adler32-zlib) [ 714.773333] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 715.035033] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 716.768128] Key type lgssc registered [ 717.866427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 728.924984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 728.962270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.658077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.315868] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 740.839971] LustreError: 24281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.875218] LustreError: 24281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 745.955774] LustreError: 24282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 746.573888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 747.056246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 752.291182] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 761.552553] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 768.298451] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 776.354368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 776.786114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 782.907842] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 792.606308] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 801.405836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 812.013227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.016291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 812.039940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 816.577547] Lustre: server umount lustre-OST0000 complete [ 822.246909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 822.280047] Lustre: Skipped 1 previous similar message [ 822.296818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 822.313374] Lustre: Skipped 2 previous similar messages [ 827.361823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 828.366500] Lustre: server umount lustre-MDT0000 complete [ 831.462688] LustreError: 25245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 831.507907] LustreError: 25245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 834.295516] LustreError: 24263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963969 with bad export cookie 9223188678966720866 [ 834.303444] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 834.306446] LustreError: 24263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 835.424935] Lustre: server umount lustre-MDT0001 complete [ 852.153893] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 855.748736] Key type lgssc unregistered [ 856.242393] LNet: 28258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.256564] LNetError: 28258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.279932] LNet: Removed LNI 192.168.204.111@tcp [ 857.347374] Key type .llcrypt unregistered [ 857.352151] Key type ._llcrypt unregistered [ 885.082541] Key type ._llcrypt registered [ 885.085911] Key type .llcrypt registered [ 885.212288] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 886.687617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.730178] alg: No test for adler32 (adler32-zlib) [ 888.060181] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 888.297336] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 889.968378] Key type lgssc registered [ 891.887371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 901.348355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 901.361095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 902.751347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.427574] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 912.871840] LustreError: 29672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 912.892323] LustreError: 29672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 916.909371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.336041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 922.474620] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 932.388929] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 939.488569] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 956.726044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 957.158522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.407925] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 972.543810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 973.913512] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 983.015730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.017750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 983.026268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 983.038704] Lustre: Skipped 1 previous similar message [ 983.073501] LustreError: Skipped 1 previous similar message [ 987.572989] Lustre: server umount lustre-OST0000 complete [ 993.254797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.284531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.292627] Lustre: Skipped 2 previous similar messages [ 997.921646] Lustre: server umount lustre-MDT0000 complete [ 998.370126] LustreError: 29668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.382239] LustreError: 29668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1002.474038] LustreError: 29671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.492663] LustreError: 29671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1002.683344] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964138 with bad export cookie 7603377593878959444 [ 1002.688800] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.696419] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1003.191623] Lustre: server umount lustre-MDT0001 complete [ 1015.414544] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1018.687772] Key type lgssc unregistered [ 1019.134635] LNet: 33564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.149147] LNetError: 33564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.174609] LNet: Removed LNI 192.168.204.111@tcp [ 1020.350504] Key type .llcrypt unregistered [ 1020.354751] Key type ._llcrypt unregistered [ 1045.414551] Key type ._llcrypt registered [ 1045.417197] Key type .llcrypt registered [ 1045.518642] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1046.903418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1046.960870] alg: No test for adler32 (adler32-zlib) [ 1048.130161] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1048.471215] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1050.264304] Key type lgssc registered [ 1051.493269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1059.528824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1059.586595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.249254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1066.839436] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1070.572028] LustreError: 34971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1070.586401] LustreError: 34971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1075.683948] LustreError: 34972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.706707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.109823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1082.122658] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1091.182459] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1097.651516] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1104.964623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.377982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.470232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1112.009285] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1121.551449] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1133.025574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1133.042782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.066162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.136703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1137.139630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.157669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.183571] Lustre: Skipped 1 previous similar message [ 1138.799709] Lustre: server umount lustre-OST0000 complete [ 1143.778767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1143.786574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.793994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.367789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.383059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.482117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1152.488786] Lustre: Skipped 2 previous similar messages [ 1157.088147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1157.223249] Lustre: server umount lustre-MDT0000 complete [ 1160.428300] LustreError: 34952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964295 with bad export cookie 17056359284704599178 [ 1160.469928] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.728649] Lustre: server umount lustre-MDT0001 complete [ 1171.731372] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1174.321924] Key type lgssc unregistered [ 1174.631750] LNet: 38507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.644828] LNetError: 38507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1174.668857] LNet: Removed LNI 192.168.204.111@tcp [ 1175.394254] Key type .llcrypt unregistered [ 1175.397385] Key type ._llcrypt unregistered [ 1198.148393] Key type ._llcrypt registered [ 1198.151916] Key type .llcrypt registered [ 1198.231292] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1198.973464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1199.141174] alg: No test for adler32 (adler32-zlib) [ 1200.287514] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1200.513380] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1202.192243] Key type lgssc registered [ 1203.806626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1213.541558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1213.552987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.913967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.373284] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1225.189040] LustreError: 39917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1225.217844] LustreError: 39917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1228.196871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.586948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1233.065914] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1240.470973] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1245.613486] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1251.822474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1252.081356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.805175] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1260.276096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1264.735883] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1272.854899] Lustre: lustre-OST0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 1274.850850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1274.857510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.866267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1275.878785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1275.880206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.899042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1277.652639] Lustre: server umount lustre-OST0000 complete [ 1296.357228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.367795] Lustre: Skipped 1 previous similar message [ 1296.369640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.378532] Lustre: Skipped 1 previous similar message [ 1300.515787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1300.523650] Lustre: Skipped 1 previous similar message [ 1302.660071] Lustre: server umount lustre-MDT0000 complete [ 1305.570095] LustreError: 41157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1305.583724] LustreError: 41157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1306.308365] LustreError: 42621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964441 with bad export cookie 15815536900638637176 [ 1306.313261] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1306.325198] LustreError: 42621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1312.632633] Lustre: server umount lustre-MDT0001 complete [ 1322.870533] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1325.790440] Key type lgssc unregistered [ 1326.055426] LNet: 43625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.063381] LNetError: 43625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.084145] LNet: Removed LNI 192.168.204.111@tcp [ 1326.801710] Key type .llcrypt unregistered [ 1326.804380] Key type ._llcrypt unregistered [ 1350.722902] Key type ._llcrypt registered [ 1350.726297] Key type .llcrypt registered [ 1350.841809] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1351.846935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1351.912347] alg: No test for adler32 (adler32-zlib) [ 1353.053424] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1353.358853] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1355.096940] Key type lgssc registered [ 1356.113023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1363.718727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1363.735415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.180872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.037408] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1374.700520] LustreError: 45032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.734584] LustreError: 45032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1379.814196] LustreError: 45033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.954877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.626961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1386.875487] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1394.227173] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1399.772663] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1405.393177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1405.697685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.797237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1410.547855] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1416.751178] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1424.421516] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 1425.391294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.409921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1425.413403] Lustre: Skipped 2 previous similar messages [ 1425.430308] Lustre: Skipped 2 previous similar messages [ 1429.500099] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 1430.273894] Lustre: server umount lustre-MDT0000 complete [ 1430.500469] LustreError: 45027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.519064] LustreError: 45027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1433.903379] LustreError: 45769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964569 with bad export cookie 4976017977621976810 [ 1433.907719] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1433.911036] LustreError: 45769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1434.624384] LustreError: 45029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.639158] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [ 1435.625567] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.625590] LustreError: 45028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.660641] LustreError: 45028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1439.741517] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [ 1439.746647] Lustre: Skipped 2 previous similar messages [ 1440.204552] Lustre: server umount lustre-MDT0001 complete [ 1460.550342] Lustre: server umount lustre-OST0000 complete [ 1477.004487] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1479.979911] Key type lgssc unregistered [ 1480.374756] LNet: 48892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.384379] LNetError: 48892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.403313] LNet: Removed LNI 192.168.204.111@tcp [ 1481.332387] Key type .llcrypt unregistered [ 1481.334614] Key type ._llcrypt unregistered [ 1505.142254] Key type ._llcrypt registered [ 1505.144779] Key type .llcrypt registered [ 1505.243859] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1506.360023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1506.522744] alg: No test for adler32 (adler32-zlib) [ 1507.708540] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1508.031531] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1509.768336] Key type lgssc registered [ 1511.125429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1520.143175] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1520.169285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.776760] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1540.896476] LustreError: 50277:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 1567.585105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.912785] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1582.532728] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1749.873411] Lustre: server umount lustre-OST0000 complete [ 1771.529376] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1775.012664] Key type lgssc unregistered [ 1775.432762] LNet: 51953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.449797] LNetError: 51953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.477242] LNet: Removed LNI 192.168.204.111@tcp [ 1776.558228] Key type .llcrypt unregistered [ 1776.563254] Key type ._llcrypt unregistered [ 1803.417742] Key type ._llcrypt registered [ 1803.420921] Key type .llcrypt registered [ 1803.547810] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1804.676921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1804.690450] alg: No test for adler32 (adler32-zlib) [ 1805.801837] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1806.037454] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1807.675051] Key type lgssc registered [ 1809.137967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1818.805892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1818.831362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.560115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.364528] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1830.881573] LustreError: 53367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.890109] LustreError: 53367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1836.005884] LustreError: 53368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1836.459093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.717671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1841.701256] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1851.358689] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1858.558455] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1868.516078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1868.816085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.639292] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1886.729862] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1890.670158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1890.673039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.831855] Lustre: server umount lustre-OST0000 complete [ 1901.025582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.027226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.040490] Lustre: Skipped 1 previous similar message [ 1901.057688] Lustre: Skipped 2 previous similar messages [ 1905.549277] Lustre: server umount lustre-MDT0000 complete [ 1906.145670] LustreError: 53368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.167742] LustreError: 53368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1909.170095] LustreError: 53347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965044 with bad export cookie 1422678419015228259 [ 1909.187016] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1909.193788] LustreError: 53347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1910.149878] Lustre: server umount lustre-MDT0001 complete [ 1926.402195] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1930.060194] Key type lgssc unregistered [ 1930.401049] LNet: 56897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.415210] LNetError: 56897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.446400] LNet: Removed LNI 192.168.204.111@tcp [ 1931.309076] Key type .llcrypt unregistered [ 1931.310618] Key type ._llcrypt unregistered [ 1960.301138] Key type ._llcrypt registered [ 1960.304296] Key type .llcrypt registered [ 1960.457934] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1961.569684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1961.603139] alg: No test for adler32 (adler32-zlib) [ 1962.858898] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1963.159141] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1964.848301] Key type lgssc registered [ 1966.238216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1975.541192] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1975.560067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1991.200488] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1996.320468] LustreError: 58280:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 2023.013748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.718860] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2041.168458] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2049.789272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.714109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.818166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2056.174553] LustreError: 58305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2056.202442] LustreError: 58305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2056.740611] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2058.214764] LustreError: 58306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.230058] LustreError: 58306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2063.334894] LustreError: 59316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.358468] LustreError: 59316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2068.220616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.454893] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2068.633610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2074.483992] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2084.706334] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2091.783644] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2094.566211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.579499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2094.591598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2094.600275] Lustre: Skipped 2 previous similar messages [ 2099.688879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2099.691638] Lustre: Skipped 1 previous similar message [ 2104.805909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2104.812612] Lustre: Skipped 1 previous similar message [ 2108.384130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2108.616928] Lustre: server umount lustre-OST0000 complete [ 2135.011648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2135.024943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.044675] Lustre: Skipped 1 previous similar message [ 2135.057329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2140.566484] Lustre: server umount lustre-MDT0000 complete [ 2141.666264] LustreError: 59317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.695245] LustreError: 59317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2145.183671] LustreError: 58287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965280 with bad export cookie 16538182904774413996 [ 2145.207944] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2145.212771] LustreError: 58287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2151.679190] Lustre: server umount lustre-MDT0001 complete [ 2164.702133] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2167.930396] Key type lgssc unregistered [ 2168.384160] LNet: 62030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.410967] LNetError: 62030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.445605] LNet: Removed LNI 192.168.204.111@tcp [ 2169.377558] Key type .llcrypt unregistered [ 2169.381313] Key type ._llcrypt unregistered [ 2197.253495] Key type ._llcrypt registered [ 2197.262444] Key type .llcrypt registered [ 2197.451423] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2198.940303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2199.369819] alg: No test for adler32 (adler32-zlib) [ 2200.476886] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 2200.814838] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2202.544551] Key type lgssc registered [ 2204.021541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2213.434236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2213.460981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.839655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.037745] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2225.127210] LustreError: 63446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.154146] LustreError: 63446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2229.379277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.725991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2234.141382] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2243.297493] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2250.787155] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2259.190262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2259.707994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.730357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2267.313251] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2275.909457] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2295.780888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.785827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2295.809764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2295.829254] Lustre: Skipped 1 previous similar message [ 2300.904699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2300.914886] Lustre: Skipped 1 previous similar message [ 2301.607705] Lustre: server umount lustre-OST0000 complete [ 2306.018227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.032285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2306.032904] Lustre: Skipped 2 previous similar messages [ 2306.065281] Lustre: Skipped 1 previous similar message [ 2311.137879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.142017] Lustre: Skipped 1 previous similar message [ 2316.259210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.268517] Lustre: Skipped 1 previous similar message [ 2320.352875] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2320.733716] Lustre: server umount lustre-MDT0000 complete [ 2321.377741] LustreError: 63446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.410101] LustreError: 63446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2325.153036] LustreError: 63428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965460 with bad export cookie 12068571442098688748 [ 2325.176276] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2325.600390] Lustre: server umount lustre-MDT0001 complete [ 2339.979322] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2343.316264] Key type lgssc unregistered [ 2343.504934] LNet: 67003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.510294] LNetError: 67003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2343.528592] LNet: Removed LNI 192.168.204.111@tcp [ 2344.347281] Key type .llcrypt unregistered [ 2344.353232] Key type ._llcrypt unregistered [ 2376.101053] Key type ._llcrypt registered [ 2376.102711] Key type .llcrypt registered [ 2376.298231] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2377.872579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2377.971933] alg: No test for adler32 (adler32-zlib) [ 2379.325709] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 2379.675821] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2381.400171] Key type lgssc registered [ 2383.085893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2398.513599] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 14:25:32 (1790965532) [ 2407.425599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2407.464833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.956429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.405727] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2419.169526] LustreError: 68716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.194133] LustreError: 68716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2420.210365] LustreError: 68715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.360149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.936763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2426.992943] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2434.261242] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2439.547991] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2446.162527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.448371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2453.794213] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2454.648574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2463.393782] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2469.652988] Lustre: Failing over lustre-MDT0000 [ 2469.847631] Lustre: server umount lustre-MDT0000 complete [ 2470.372753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.375398] LustreError: 69450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.402365] Lustre: Skipped 1 previous similar message [ 2475.492125] LustreError: 69443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.498931] LustreError: 69443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2479.280047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.410748] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.603706] Lustre: *** cfs_fail_loc=135, val=0*** [ 2479.659736] LustreError: 71800:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2479.665511] LustreError: 71800:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -2 [ 2479.670455] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2479.670455] [ 2479.675738] LustreError: MGC192.168.204.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2479.686248] LustreError: 71789:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2479.691279] LustreError: 71789:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2479.695255] LustreError: 71789:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2479.697821] LustreError: 71789:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2479.713847] Lustre: server umount lustre-MDT0000 complete [ 2479.717659] LustreError: 71789:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2480.611221] LustreError: 68710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2480.648216] LustreError: 68710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2486.645115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.812432] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2486.822566] LustreError: 68707:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2487.071728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.118848] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2487.367813] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2492.448349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2492.542586] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2492.614650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2492.823795] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2504.163030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2504.180975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.205234] Lustre: Skipped 1 previous similar message [ 2504.219457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.747738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.759207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.595905] Lustre: server umount lustre-OST0000 complete [ 2517.991536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.999413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2518.039033] Lustre: Skipped 1 previous similar message [ 2518.065658] Lustre: Skipped 2 previous similar messages [ 2520.409673] Lustre: server umount lustre-MDT0000 complete [ 2523.107191] LustreError: 68710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.131483] LustreError: 68710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2524.933392] LustreError: 68696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965660 with bad export cookie 5182907319472673971 [ 2524.938326] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2524.955460] LustreError: 68696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2525.656332] Lustre: server umount lustre-MDT0001 complete [ 2540.281160] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2543.210237] Key type lgssc unregistered [ 2543.573313] LNet: 73581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2543.582647] LNetError: 73581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2543.593920] LNet: Removed LNI 192.168.204.111@tcp [ 2544.586578] Key type .llcrypt unregistered [ 2544.593869] Key type ._llcrypt unregistered [ 2573.732666] Key type ._llcrypt registered [ 2573.743796] Key type .llcrypt registered [ 2573.983258] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2575.567712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2575.620953] alg: No test for adler32 (adler32-zlib) [ 2576.916219] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 2577.456650] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2579.232222] Key type lgssc registered [ 2580.364790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2589.382872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2589.403286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.787681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.340743] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2600.937712] LustreError: 74987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.954205] LustreError: 74987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2605.180848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.554847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2610.108948] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2617.497080] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2623.569719] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2631.130629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2631.501467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.645579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2638.064567] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2646.463652] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2656.268197] Lustre: Failing over lustre-MDT0000 [ 2656.557995] Lustre: server umount lustre-MDT0000 complete [ 2656.738535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2656.753337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.766749] LustreError: 74988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.886777] LustreError: 74983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.368738] LustreError: 75717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.376255] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.410528] LustreError: 75717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2661.459745] Lustre: Skipped 1 previous similar message [ 2666.466624] LustreError: 74987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.490108] LustreError: 74987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2669.162737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.362797] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2669.540678] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2669.697657] LustreError: 78075:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2669.706628] LustreError: 78075:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -9 [ 2669.715217] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2669.715217] [ 2669.727508] LustreError: MGC192.168.204.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2669.763412] LustreError: 78063:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2669.786052] LustreError: 78063:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2669.801717] LustreError: 78063:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2669.823714] LustreError: 78063:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2669.853702] Lustre: server umount lustre-MDT0000 complete [ 2669.856456] LustreError: 78063:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2675.270621] LustreError: 74984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.299101] LustreError: 74984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2676.673729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.853332] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2677.229768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.272979] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2679.292686] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2682.252784] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2682.402317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2682.446841] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2682.544316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2692.068960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2692.082108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.105525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2697.408480] Lustre: server umount lustre-OST0000 complete [ 2702.820801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2702.830991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2702.850266] Lustre: Skipped 1 previous similar message [ 2702.863753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2702.872055] Lustre: Skipped 2 previous similar messages [ 2707.875765] Lustre: server umount lustre-MDT0000 complete [ 2707.939825] LustreError: 76545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.962580] LustreError: 76545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2712.443221] LustreError: 74968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965848 with bad export cookie 6749234799922612136 [ 2712.449977] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2713.052248] Lustre: server umount lustre-MDT0001 complete [ 2727.077521] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2729.892032] Key type lgssc unregistered [ 2730.194948] LNet: 79856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2730.200863] LNetError: 79856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2730.219837] LNet: Removed LNI 192.168.204.111@tcp [ 2731.022326] Key type .llcrypt unregistered [ 2731.024400] Key type ._llcrypt unregistered [ 2743.178635] Key type ._llcrypt registered [ 2743.186480] Key type .llcrypt registered [ 2743.340738] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2743.649466] Key type .llcrypt unregistered [ 2743.657237] Key type ._llcrypt unregistered [ 2767.512870] Key type ._llcrypt registered [ 2767.516820] Key type .llcrypt registered [ 2767.632618] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2768.824467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2768.882535] alg: No test for adler32 (adler32-zlib) [ 2770.106586] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 2770.364416] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2772.072172] Key type lgssc registered [ 2773.435763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2781.241848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2781.266113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.685599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2787.067929] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2792.935155] LustreError: 81752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.957993] LustreError: 81752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2796.336830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.780865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2801.588319] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2809.421577] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2815.329567] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2822.926344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2823.324374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2828.423720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2830.194779] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2838.870711] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2852.322917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2852.335823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.339675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2852.369094] Lustre: Skipped 1 previous similar message [ 2857.489393] Lustre: server umount lustre-OST0000 complete [ 2862.562073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.574453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.580880] Lustre: Skipped 2 previous similar messages [ 2867.684546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2867.693481] Lustre: Skipped 1 previous similar message [ 2872.806275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2876.384125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2876.816994] Lustre: server umount lustre-MDT0000 complete [ 2877.921117] LustreError: 81753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.954501] LustreError: 81753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2881.002796] LustreError: 81734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966016 with bad export cookie 14473436479872763957 [ 2881.014270] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2881.014901] LustreError: 81734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2881.294608] Lustre: server umount lustre-MDT0001 complete [ 2891.600996] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 2894.043855] Key type lgssc unregistered [ 2894.295536] LNet: 85299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2894.307133] LNetError: 85299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2894.330856] LNet: Removed LNI 192.168.204.111@tcp [ 2895.298343] Key type .llcrypt unregistered [ 2895.301102] Key type ._llcrypt unregistered [ 2919.088615] Key type ._llcrypt registered [ 2919.094923] Key type .llcrypt registered [ 2919.235502] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2920.238450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2920.338035] alg: No test for adler32 (adler32-zlib) [ 2921.517896] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 2921.768670] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2923.440428] Key type lgssc registered [ 2924.871843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2935.254758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2935.270222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.778572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.895479] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2947.050523] LustreError: 86710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.077344] LustreError: 86710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2950.729097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2951.232039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2955.724315] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2964.264496] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2971.510915] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2979.142753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2979.616258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.700678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2986.559308] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2994.538886] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3001.313748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3001.327400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.341637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3002.341274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3002.353634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.365846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3002.813736] Lustre: server umount lustre-OST0000 complete [ 3007.457893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3007.471240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.494360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3009.506865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.508952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3009.527154] Lustre: Skipped 1 previous similar message [ 3013.768675] Lustre: server umount lustre-MDT0000 complete [ 3014.634717] LustreError: 86710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3014.670454] LustreError: 86710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3018.733337] LustreError: 86709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.781148] LustreError: 86709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3018.900508] LustreError: 86688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966154 with bad export cookie 10516335423149872320 [ 3018.914829] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3019.232363] Lustre: server umount lustre-MDT0001 complete [ 3031.844872] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3035.085674] Key type lgssc unregistered [ 3035.426421] LNet: 90244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3035.447225] LNetError: 90244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3035.464238] LNet: Removed LNI 192.168.204.111@tcp [ 3036.431211] Key type .llcrypt unregistered [ 3036.434821] Key type ._llcrypt unregistered [ 3063.179078] Key type ._llcrypt registered [ 3063.183333] Key type .llcrypt registered [ 3063.380854] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3063.920031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3063.940169] alg: No test for adler32 (adler32-zlib) [ 3064.990209] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3065.261588] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3066.953739] Key type lgssc registered [ 3068.077690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3075.509360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3075.528584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.864112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.376952] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3087.338489] LustreError: 91657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3087.368565] LustreError: 91657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3089.313114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.843562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3094.369168] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3101.606331] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3107.346951] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3114.605826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3114.906194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3121.046426] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3122.112602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3128.569173] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3147.752474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3147.773438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.805613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3150.819217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3150.822314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.154558] Lustre: server umount lustre-OST0000 complete [ 3161.061086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.086573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3161.094973] Lustre: Skipped 1 previous similar message [ 3163.473145] Lustre: server umount lustre-MDT0000 complete [ 3166.182060] LustreError: 94099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3166.208305] LustreError: 94099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3167.632930] LustreError: 91638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966303 with bad export cookie 837471399978034085 [ 3167.634414] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3167.655534] LustreError: 91638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3167.970068] Lustre: server umount lustre-MDT0001 complete [ 3177.827353] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3180.299353] Key type lgssc unregistered [ 3180.590913] LNet: 95211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3180.598861] LNetError: 95211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3180.626960] LNet: Removed LNI 192.168.204.111@tcp [ 3181.284792] Key type .llcrypt unregistered [ 3181.287950] Key type ._llcrypt unregistered [ 3207.788396] Key type ._llcrypt registered [ 3207.790539] Key type .llcrypt registered [ 3207.949559] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3209.551286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3209.603892] alg: No test for adler32 (adler32-zlib) [ 3210.857967] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3211.188857] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3212.960947] Key type lgssc registered [ 3214.422863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3223.014160] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3223.060979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3238.624447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3243.744337] LustreError: 96594:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 3270.496290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.360682] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3287.678864] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3294.712972] Lustre: server umount lustre-OST0000 complete [ 3305.875854] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 14:40:40 (1790966440) [ 3315.931400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.373978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.921235] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3321.829576] LustreError: 98233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.857963] LustreError: 98233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3325.939788] LustreError: 98234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.052442] LustreError: 98233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.993774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.463709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3337.645262] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3347.604140] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3355.286557] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3365.316880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3365.741137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3373.663288] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3383.679909] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3386.233215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3401.701157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.747645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3401.748018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3405.720874] Lustre: server umount lustre-OST0000 complete [ 3411.945964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.953694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3411.967186] Lustre: Skipped 2 previous similar messages [ 3411.978594] Lustre: Skipped 2 previous similar messages [ 3415.870948] Lustre: server umount lustre-MDT0000 complete [ 3417.061438] LustreError: 99776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3417.084658] LustreError: 99776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3420.007329] LustreError: 98216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966555 with bad export cookie 8537269610592761215 [ 3420.008371] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3420.023542] LustreError: 98216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3420.464899] Lustre: server umount lustre-MDT0001 complete [ 3432.366743] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3435.245106] Key type lgssc unregistered [ 3435.590725] LNet: 101930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3435.600256] LNetError: 101930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3435.622573] LNet: Removed LNI 192.168.204.111@tcp [ 3436.622607] Key type .llcrypt unregistered [ 3436.628370] Key type ._llcrypt unregistered [ 3479.052535] Key type ._llcrypt registered [ 3479.055152] Key type .llcrypt registered [ 3479.185916] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3480.371661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3480.453808] alg: No test for adler32 (adler32-zlib) [ 3481.724613] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3482.107121] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3483.944404] Key type lgssc registered [ 3485.499834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3494.724904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3494.744391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.337761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.375530] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3505.657834] LustreError: 104107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3505.678877] LustreError: 104107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3510.762594] LustreError: 104106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.942530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3515.734599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3520.517384] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3528.655266] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3535.540488] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3544.263978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3544.980588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.250407] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3563.724028] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3565.516672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3576.817188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3576.823969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3576.835409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3580.901303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3580.901798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3580.911352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.033239] Lustre: server umount lustre-OST0000 complete [ 3587.553512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3587.563707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.587535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3587.599517] Lustre: Skipped 1 previous similar message [ 3591.137942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3591.138637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3592.965275] Lustre: server umount lustre-MDT0000 complete [ 3596.260757] LustreError: 104106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3596.291031] LustreError: 104106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3597.977136] LustreError: 104086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966733 with bad export cookie 2960737573706082910 [ 3597.986433] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3597.994311] LustreError: 104086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3598.607468] Lustre: server umount lustre-MDT0001 complete [ 3612.662961] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3617.389906] Key type lgssc unregistered [ 3617.842714] LNet: 107665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3617.859925] LNetError: 107665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3617.893061] LNet: Removed LNI 192.168.204.111@tcp [ 3619.609907] Key type .llcrypt unregistered [ 3619.618318] Key type ._llcrypt unregistered [ 3640.588853] Key type ._llcrypt registered [ 3640.592467] Key type .llcrypt registered [ 3640.723719] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3641.630090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3641.679911] alg: No test for adler32 (adler32-zlib) [ 3642.947283] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3643.300800] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3645.040405] Key type lgssc registered [ 3646.218906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3655.601115] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3655.633737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3671.201804] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3676.320641] LustreError: 108906:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 3703.072244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3712.043868] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3723.944714] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3733.682057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3734.123799] LustreError: MGC192.168.204.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3734.132724] LustreError: 109932:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3734.137740] LustreError: 109932:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3734.144888] LustreError: 109932:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3734.215953] Lustre: server umount lustre-MDT0000 complete [ 3734.218816] LustreError: 109932:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3753.440861] Lustre: 108241:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790966873/real 1790966873] req@00000000995b0f35 x1877964780211584/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790966889 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3753.488371] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3753.495674] LustreError: 108918:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3774.100335] Lustre: server umount lustre-OST0000 complete [ 3787.365525] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3802.504095] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3847.170619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.665514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3847.741669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3847.921372] Lustre: lustre-MDT0000: new disk, initializing [ 3848.119088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3848.178942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3853.954445] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3866.412382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.530881] Lustre: 112749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3866.600727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3866.605121] Lustre: Skipped 1 previous similar message [ 3866.742964] Lustre: lustre-MDT0001: new disk, initializing [ 3866.930448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3867.011765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3867.030192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3872.773961] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3884.534745] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3892.785365] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3902.320793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3902.735381] Lustre: lustre-OST0000: new disk, initializing [ 3902.747296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3902.760684] Lustre: 114720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3902.882088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.223592] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3911.754154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3911.767838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3911.888532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3920.759759] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3928.833978] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3929.389457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3935.882653] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3936.095408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3942.887898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.888610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3942.891892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3942.914952] Lustre: Skipped 1 previous similar message [ 3942.929029] LustreError: Skipped 1 previous similar message [ 3944.821557] Lustre: server umount lustre-OST0000 complete [ 3953.121237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.143037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3953.163634] Lustre: Skipped 1 previous similar message [ 3953.174891] Lustre: Skipped 2 previous similar messages [ 3955.765402] Lustre: server umount lustre-MDT0000 complete [ 3959.992222] LustreError: 112742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967095 with bad export cookie 12225154000599959919 [ 3959.996760] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3960.015482] LustreError: 112742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3960.575471] Lustre: server umount lustre-MDT0001 complete [ 3971.429894] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 14:51:45 (1790967105) [ 3976.498697] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4001.977841] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 4012.754920] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4053.649359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.056541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4054.079684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4054.173581] Lustre: lustre-MDT0000: new disk, initializing [ 4054.388801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.426895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4059.249942] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4070.235340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.301565] Lustre: 119626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4070.352692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4070.355533] Lustre: Skipped 1 previous similar message [ 4070.406825] Lustre: lustre-MDT0001: new disk, initializing [ 4070.474352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4070.513106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4070.529727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4076.885762] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4093.392616] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4101.118199] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4110.164934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4110.615104] Lustre: lustre-OST0000: new disk, initializing [ 4110.618305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4110.622822] Lustre: 121595:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4110.734308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4117.613513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4117.624732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4117.774379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4118.998650] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4132.238912] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4138.710458] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4139.120686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4146.188811] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4146.654060] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4153.829315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4153.833937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.862251] LustreError: Skipped 1 previous similar message [ 4153.862992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4153.872453] Lustre: Skipped 1 previous similar message [ 4153.926880] Lustre: Skipped 1 previous similar message [ 4156.395850] Lustre: server umount lustre-OST0000 complete [ 4162.530450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4162.548520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.579061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4169.194732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4169.199626] Lustre: Skipped 3 previous similar messages [ 4176.352230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4176.623888] Lustre: server umount lustre-MDT0000 complete [ 4179.434732] LustreError: 119637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4179.466734] LustreError: 119637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4180.851394] LustreError: 119617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967316 with bad export cookie 12225154000599960801 [ 4180.855277] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4180.876573] LustreError: 119617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4181.375793] Lustre: server umount lustre-MDT0001 complete [ 4190.701491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.333684] LustreError: 123816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4191.504143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.296817] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4196.835237] LustreError: 123817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.931334] LustreError: 123816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.004549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.146078] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4218.951499] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4224.175476] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4234.450610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4240.926286] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4248.673630] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4256.177472] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4265.465245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.482609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4265.484258] Lustre: Skipped 1 previous similar message [ 4265.492383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4265.492395] Lustre: Skipped 2 previous similar messages [ 4267.854500] Lustre: server umount lustre-OST0000 complete [ 4275.684641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4275.726565] Lustre: Skipped 1 previous similar message [ 4281.385567] Lustre: server umount lustre-MDT0000 complete [ 4285.928310] LustreError: 123812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4285.968474] LustreError: 123812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4287.556501] LustreError: 123796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967423 with bad export cookie 12225154000599961683 [ 4287.568963] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4287.569180] LustreError: 123796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4288.117309] Lustre: server umount lustre-MDT0001 complete [ 4300.905770] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4303.763772] Key type lgssc unregistered [ 4304.247912] LNet: 127445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4304.261575] LNetError: 127445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4304.304204] LNet: Removed LNI 192.168.204.111@tcp [ 4305.358110] Key type .llcrypt unregistered [ 4305.360246] Key type ._llcrypt unregistered [ 4326.569907] Key type ._llcrypt registered [ 4326.572287] Key type .llcrypt registered [ 4326.684789] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 4342.615817] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4343.651567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4343.731162] alg: No test for adler32 (adler32-zlib) [ 4344.877422] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 4345.231167] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4347.024177] Key type lgssc registered [ 4348.889764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4393.233695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4393.277511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4394.505947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4394.559460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4394.679552] Lustre: lustre-MDT0000: new disk, initializing [ 4394.807342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4394.844166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4399.882791] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4411.687820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.926057] Lustre: 130598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4412.036670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4412.047054] Lustre: Skipped 1 previous similar message [ 4412.218634] Lustre: lustre-MDT0001: new disk, initializing [ 4412.412861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4412.480831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4412.496746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4417.594628] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4428.002247] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4436.410304] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4448.286158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4448.982717] Lustre: lustre-OST0000: new disk, initializing [ 4449.007062] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4449.019981] Lustre: 132565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4449.263687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.975469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4455.990607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4456.177780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4457.710396] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4466.973931] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4473.073289] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4473.274590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4480.519661] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4480.746674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4487.145736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4487.150253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4487.185598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4487.201183] LustreError: Skipped 1 previous similar message [ 4487.214658] Lustre: Skipped 1 previous similar message [ 4492.281142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4492.286880] Lustre: Skipped 1 previous similar message [ 4497.382217] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4497.390815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4497.408487] Lustre: Skipped 1 previous similar message [ 4497.688616] Lustre: server umount lustre-OST0000 complete [ 4502.499181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4502.504588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.521309] Lustre: Skipped 2 previous similar messages [ 4502.544323] Lustre: Skipped 1 previous similar message [ 4507.619283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4507.633836] Lustre: Skipped 1 previous similar message [ 4508.660844] Lustre: server umount lustre-MDT0000 complete [ 4512.906781] LustreError: 130589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967648 with bad export cookie 16125856495476656350 [ 4512.916062] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4512.924529] LustreError: 130589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4513.295884] Lustre: server umount lustre-MDT0001 complete [ 4522.810841] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 15:00:57 (1790967657) [ 4530.011138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4530.463609] LustreError: 134989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4530.498742] LustreError: 134989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4530.567480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4535.783247] LustreError: 134990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4536.047425] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4540.897465] LustreError: 134989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.945274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4546.018137] LustreError: 134990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.286157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4550.814636] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4557.978948] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4566.059020] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4571.120518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4571.200737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4571.226773] Lustre: Skipped 1 previous similar message [ 4576.660300] Lustre: server umount lustre-MDT0000 complete [ 4581.352624] LustreError: 134989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.364698] LustreError: 134989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4582.929304] LustreError: 134971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967718 with bad export cookie 16125856495476657232 [ 4582.937828] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4582.945833] LustreError: 134971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4583.460929] Lustre: server umount lustre-MDT0001 complete [ 4595.124848] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 15:02:09 (1790967729) [ 4603.653025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4619.362571] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4624.481748] LustreError: 137405:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 4650.272269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4655.023608] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4662.627814] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4664.269269] Lustre: server umount lustre-OST0000 complete [ 4672.424610] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 15:03:27 (1790967807) [ 4681.790879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.892403] LustreError: 138853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4682.965083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4689.596311] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4701.574172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.958406] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4716.944609] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4724.873941] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4733.862624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4734.257342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4734.266886] Lustre: Skipped 1 previous similar message [ 4742.817184] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4754.870592] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4768.245143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4768.254758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4768.254979] Lustre: Skipped 3 previous similar messages [ 4768.283028] Lustre: Skipped 4 previous similar messages [ 4773.781527] Lustre: server umount lustre-MDT0000 complete [ 4778.423912] LustreError: 138835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967913 with bad export cookie 16125856495476657897 [ 4778.435871] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4778.446690] LustreError: 138835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4778.490308] LustreError: 140026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4778.498081] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4778.524208] LustreError: 140026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4778.843921] Lustre: server umount lustre-MDT0001 complete [ 4786.083773] Lustre: server umount lustre-OST0000 complete [ 4795.843587] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:05:30 (1790967930) [ 4803.321294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4803.882234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.578981] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4814.313770] LustreError: 142527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4814.338019] LustreError: 142527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4818.776413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4823.234408] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4832.007680] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4838.034174] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4844.959579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4849.480466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4851.746287] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4863.471436] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4869.848897] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4870.280509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4876.237661] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4876.815289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4879.334532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4879.334687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.343384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4879.352988] Lustre: Skipped 1 previous similar message [ 4879.375891] Lustre: Skipped 4 previous similar messages [ 4885.373668] Lustre: server umount lustre-OST0000 complete [ 4890.595273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4896.571528] Lustre: server umount lustre-MDT0000 complete [ 4899.815189] LustreError: 142523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4899.840070] LustreError: 142523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4900.864620] LustreError: 142507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968036 with bad export cookie 16125856495476658968 [ 4900.878022] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4900.901948] LustreError: 142507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4901.294468] Lustre: server umount lustre-MDT0001 complete [ 4911.239445] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:07:25 (1790968045) [ 4919.108843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4934.816615] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4939.937503] LustreError: 146687:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 4965.600934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4965.613257] Lustre: Skipped 2 previous similar messages [ 4972.317504] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4981.345836] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4988.528842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4988.751326] LustreError: 128816:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b7251bd4 x1877965516447872/t0(0) o250->MGC192.168.204.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4989.299731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4994.021521] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5004.929878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5010.687920] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5020.734684] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5028.267737] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5033.095183] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5033.353432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5039.218829] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5039.487826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5046.242325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5046.259265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.273567] Lustre: Skipped 2 previous similar messages [ 5046.277984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.282734] Lustre: Skipped 5 previous similar messages [ 5048.049774] Lustre: server umount lustre-MDT0000 complete [ 5051.876650] LustreError: 147724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5051.896695] LustreError: 147724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5052.373731] LustreError: 146694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968187 with bad export cookie 16125856495476659990 [ 5052.386740] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5052.402164] LustreError: 146694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5052.919527] Lustre: server umount lustre-MDT0001 complete [ 5067.298957] Lustre: server umount lustre-OST0000 complete [ 5076.621152] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 15:10:11 (1790968211) [ 5083.859886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5099.488924] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5104.618756] LustreError: 150884:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 5136.811992] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5145.223338] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5152.548261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5153.201482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5157.872867] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5166.808936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5171.735488] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5181.069306] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5187.713853] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5196.023978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5196.257363] Lustre: lustre-OST0001: new disk, initializing [ 5196.262309] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5196.268551] Lustre: 153758:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5202.280348] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5204.036230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5204.045943] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5204.178037] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5211.027344] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5215.670469] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5215.905604] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5221.199203] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5221.517081] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5224.945135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5224.957562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5224.964256] Lustre: Skipped 2 previous similar messages [ 5224.966625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5224.975425] Lustre: Skipped 3 previous similar messages [ 5229.713632] Lustre: server umount lustre-OST0000 complete [ 5235.171354] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5235.188594] LustreError: Skipped 1 previous similar message [ 5239.770423] Lustre: server umount lustre-OST0001 complete [ 5249.561611] Lustre: server umount lustre-MDT0000 complete [ 5253.454255] LustreError: 150891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968389 with bad export cookie 16125856495476660942 [ 5253.461385] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5253.758595] Lustre: server umount lustre-MDT0001 complete [ 5278.793236] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 15:13:33 (1790968413) [ 5280.712900] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5282.283671] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5284.276499] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:13:38 (1790968418) [ 5292.128663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5292.601889] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5292.678894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5293.170529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5293.186591] Lustre: Skipped 6 previous similar messages [ 5299.273573] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5310.382400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.590515] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5310.677878] Lustre: 157319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5317.658175] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5326.083799] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5332.424833] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5340.053565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5340.246316] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5345.968679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5346.897174] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5354.204576] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5358.894728] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5359.122821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5363.685433] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5363.895837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5366.241376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5366.251721] LustreError: Skipped 1 previous similar message [ 5366.261792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.281677] Lustre: Skipped 4 previous similar messages [ 5380.067040] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5380.202914] Lustre: server umount lustre-OST0000 complete [ 5399.792864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5404.823459] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5407.217317] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.231975] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5407.232036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.043317] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5416.216206] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5416.510150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5420.821871] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5421.002973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5426.186359] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff906242cf1800.ost_server_uuid 50 [ 5427.844975] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff906242cf1800.ost_server_uuid in FULL state after 0 sec [ 5428.058046] LustreError: 157325:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.11@tcp) returned error from blocking AST (req@00000000417dd810 x1877965516587008 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000727808e3/0xdfca8d0b4fcfe428 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.11@tcp remote: 0x22cb8ee457885109 expref: 5 pid: 158717 timeout: 5528 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5428.080330] LustreError: lustre-MDT0000: A client on nid 192.168.204.11@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5428.089161] LustreError: 157314:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.11@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000727808e3/0xdfca8d0b4fcfe428 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.11@tcp remote: 0x22cb8ee457885109 expref: 6 pid: 158717 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5432.805588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5432.812616] LustreError: Skipped 1 previous similar message [ 5437.828874] Lustre: server umount lustre-OST0000 complete [ 5455.329355] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5455.464501] Lustre: server umount lustre-MDT0000 complete [ 5458.242817] LustreError: 157309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968593 with bad export cookie 16125856495476662125 [ 5458.249762] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5458.259697] LustreError: 157309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5464.667761] Lustre: server umount lustre-MDT0001 complete [ 5475.498990] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5477.844317] Key type lgssc unregistered [ 5478.121643] LNet: 163432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5478.127609] LNetError: 163432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5478.154957] LNet: Removed LNI 192.168.204.111@tcp [ 5478.859312] Key type .llcrypt unregistered [ 5478.861232] Key type ._llcrypt unregistered [ 5497.356950] Key type ._llcrypt registered [ 5497.359411] Key type .llcrypt registered [ 5497.451047] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5498.385533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5498.405353] alg: No test for adler32 (adler32-zlib) [ 5499.397288] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 5499.563817] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5501.208177] Key type lgssc registered [ 5502.172703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5509.945508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5509.958543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5511.288371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.131276] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5520.885276] LustreError: 164822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.921922] LustreError: 164822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5522.983751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5523.452927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5527.560346] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5534.678108] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5539.892769] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5545.928830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5546.229289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5551.971277] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5556.483921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5559.112392] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5565.453734] Lustre: Failing over lustre-MDT0000 [ 5565.774858] Lustre: server umount lustre-MDT0000 complete [ 5566.951587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.966234] Lustre: Skipped 2 previous similar messages [ 5566.969646] LustreError: 167110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5567.001326] LustreError: 167110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5572.070104] LustreError: 164823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5572.087867] LustreError: 164823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5577.191305] LustreError: 164817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5577.224407] LustreError: 164817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5578.061568] LustreError: 164803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.11@tcp arrived at 1790968713 with bad export cookie 4276240102229211066 [ 5578.087393] LustreError: 164803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5582.309281] LustreError: 164818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5582.329930] LustreError: 164818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5583.328585] Lustre: 164133:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790968702/real 1790968702] req@00000000953889c3 x1877966727439616/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790968718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5583.358801] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5585.511618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.554767] LustreError: 166062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5592.565942] LustreError: 166062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5592.823584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5592.897926] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5597.110982] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5598.182866] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5598.194327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5599.362913] Lustre: lustre-MDT0000: Denying connection for new client dcb5f767-a47a-4ddc-9a2b-2cc7de732665 (at 192.168.204.11@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5608.422188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5608.424340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5608.440183] Lustre: Skipped 1 previous similar message [ 5613.542738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5613.548368] Lustre: Skipped 2 previous similar messages [ 5613.817097] Lustre: server umount lustre-OST0000 complete [ 5617.167120] LustreError: 168553:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5617.179863] Lustre: 167901:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5617.188660] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5617.222578] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5617.238592] LustreError: 167901:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b71d00c7 x1877966727459584/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5617.266599] LustreError: 167901:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5617.286340] LustreError: 167901:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5617.300231] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5617.536896] Lustre: server umount lustre-MDT0000 complete [ 5618.660499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.685792] LustreError: 164817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.202323] LustreError: 164803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968756 with bad export cookie 4276240102229211164 [ 5621.204152] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5621.215155] LustreError: 164803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5621.215366] LustreError: 164131:0:(import.c:751:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5623.607426] Lustre: server umount lustre-MDT0001 complete [ 5633.416202] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5636.031288] Key type lgssc unregistered [ 5636.284438] LNet: 169305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5636.303601] LNetError: 169305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5636.317614] LNet: Removed LNI 192.168.204.111@tcp [ 5637.145587] Key type .llcrypt unregistered [ 5637.151322] Key type ._llcrypt unregistered [ 5658.708871] Key type ._llcrypt registered [ 5658.710795] Key type .llcrypt registered [ 5658.783978] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5659.473068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5659.564347] alg: No test for adler32 (adler32-zlib) [ 5660.587367] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 5660.806584] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5662.456179] Key type lgssc registered [ 5663.429473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5670.078692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5670.100972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5671.471161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.375487] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5681.641068] LustreError: 170714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.659170] LustreError: 170714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5682.830964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5683.116746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5687.252229] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5693.566934] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5699.913130] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5706.379602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5706.817176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.083185] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5717.053329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5722.264601] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5730.857291] Lustre: server umount lustre-OST0000 complete [ 5737.447441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.458953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5737.470147] Lustre: Skipped 1 previous similar message [ 5737.494117] Lustre: Skipped 1 previous similar message [ 5742.562220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5742.568555] Lustre: Skipped 1 previous similar message [ 5745.738473] Lustre: server umount lustre-MDT0000 complete [ 5747.683426] Lustre: 170026:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790968867/real 1790968867] req@00000000253637aa x1877966896261888/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790968883 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5747.691624] LustreError: 170715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5747.721298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5747.766780] LustreError: 170715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5748.704148] Lustre: 170024:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790968868/real 1790968868] req@000000000d51b052 x1877966896262272/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790968884 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5750.082517] LustreError: 170694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968885 with bad export cookie 2174045221598685548 [ 5750.084061] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5750.092353] LustreError: 170694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5750.400302] Lustre: server umount lustre-MDT0001 complete [ 5760.447969] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5763.141906] Key type lgssc unregistered [ 5763.507158] LNet: 174245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5763.516908] LNetError: 174245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5763.540944] LNet: Removed LNI 192.168.204.111@tcp [ 5764.488312] Key type .llcrypt unregistered [ 5764.491128] Key type ._llcrypt unregistered [ 5795.158810] Key type ._llcrypt registered [ 5795.161914] Key type .llcrypt registered [ 5795.251107] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5796.022412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5796.079342] alg: No test for adler32 (adler32-zlib) [ 5797.100813] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 5797.258885] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5798.896356] Key type lgssc registered [ 5799.898556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5807.204387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5807.215816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.580771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5813.837425] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5818.850650] LustreError: 175946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5818.876455] LustreError: 175946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5822.987030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5823.369924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5827.395934] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5834.159875] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5839.301866] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5845.735350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5846.100575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5851.636302] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5854.325080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5858.071553] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5863.905978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5863.916751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.926022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5864.933959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5864.934120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.459786] Lustre: server umount lustre-OST0000 complete [ 5874.660060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5874.670634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.687816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5874.698864] Lustre: Skipped 1 previous similar message [ 5879.643239] Lustre: server umount lustre-MDT0000 complete [ 5880.294773] LustreError: 175945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5880.316742] LustreError: 175945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5884.065791] LustreError: 175926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969019 with bad export cookie 13598532409999403958 [ 5884.077598] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5884.594857] Lustre: server umount lustre-MDT0001 complete [ 5896.552268] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5898.959715] Key type lgssc unregistered [ 5899.219478] LNet: 179478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5899.232062] LNetError: 179478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5899.253513] LNet: Removed LNI 192.168.204.111@tcp [ 5900.270535] Key type .llcrypt unregistered [ 5900.272677] Key type ._llcrypt unregistered [ 5920.764589] Key type ._llcrypt registered [ 5920.767020] Key type .llcrypt registered [ 5920.838413] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5921.653899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5921.667157] alg: No test for adler32 (adler32-zlib) [ 5922.806308] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 5923.058059] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5924.752223] Key type lgssc registered [ 5925.848423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5936.492246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5936.507293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5937.856860] Lustre: *** cfs_fail_loc=135, val=0*** [ 5937.984563] LustreError: 181008:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5937.993077] LustreError: 181008:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -2 [ 5938.001762] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5938.001762] [ 5938.008541] LustreError: MGC192.168.204.111@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5938.018359] LustreError: 180969:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5938.029497] LustreError: 180969:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5938.039389] LustreError: 180969:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5938.046763] LustreError: 180969:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5938.145070] Lustre: server umount lustre-MDT0000 complete [ 5938.152155] LustreError: 180969:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5944.773491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5945.226312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5949.442310] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5954.531915] LustreError: 181379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.547629] LustreError: 181379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5957.264513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5957.581449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5961.993526] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5969.202345] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5973.413931] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5978.082755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5978.093516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5978.103987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5980.130335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5980.130940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.048075] Lustre: server umount lustre-MDT0000 complete [ 5983.216093] LustreError: 181376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5983.241846] LustreError: 181376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5985.942855] LustreError: 182948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969121 with bad export cookie 15799414169659095093 [ 5985.948806] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5985.949210] LustreError: 182948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5986.396291] Lustre: server umount lustre-MDT0001 complete [ 5998.362200] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6001.171697] Key type lgssc unregistered [ 6001.566576] LNet: 183698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6001.573367] LNetError: 183698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6001.596711] LNet: Removed LNI 192.168.204.111@tcp [ 6002.828619] Key type .llcrypt unregistered [ 6002.834595] Key type ._llcrypt unregistered [ 6025.988185] Key type ._llcrypt registered [ 6025.991571] Key type .llcrypt registered [ 6026.154902] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6026.527766] Key type .llcrypt unregistered [ 6026.533833] Key type ._llcrypt unregistered [ 6043.132526] Key type ._llcrypt registered [ 6043.135185] Key type .llcrypt registered [ 6043.229044] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6044.147036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6044.279558] alg: No test for adler32 (adler32-zlib) [ 6045.421369] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 6045.641723] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 6047.344231] Key type lgssc registered [ 6048.295531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6054.975661] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6055.009254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6070.563862] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6075.680956] LustreError: 185825:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 6102.434843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.858879] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6117.342462] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6125.041990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.612489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.720246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6130.664780] LustreError: 185850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.698162] LustreError: 185850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6130.837415] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6135.779222] LustreError: 185851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.847340] LustreError: 186861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.867122] LustreError: 186861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6139.329649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.638222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6143.797157] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6152.778313] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6160.267683] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6164.146511] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6176.737487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6176.743298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6176.755094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6180.841167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6180.844791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6180.849511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.890269] Lustre: Skipped 1 previous similar message [ 6182.243565] Lustre: server umount lustre-OST0000 complete [ 6191.076113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6191.078554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6191.098621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6191.127573] Lustre: Skipped 1 previous similar message [ 6192.560145] Lustre: server umount lustre-MDT0000 complete [ 6196.194702] LustreError: 186861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6196.212468] LustreError: 186861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6196.404547] LustreError: 185833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969331 with bad export cookie 16811682896650149173 [ 6196.408136] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6196.414988] LustreError: 185833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6196.830834] Lustre: server umount lustre-MDT0001 complete [ 6208.017389] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6210.736988] Key type lgssc unregistered [ 6211.049206] LNet: 189839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6211.061372] LNetError: 189839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6211.094415] LNet: Removed LNI 192.168.204.111@tcp [ 6212.001272] Key type .llcrypt unregistered [ 6212.007894] Key type ._llcrypt unregistered [ 6235.880612] Key type ._llcrypt registered [ 6235.883947] Key type .llcrypt registered [ 6236.100396] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6237.270991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6237.412839] alg: No test for adler32 (adler32-zlib) [ 6238.500134] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 6238.739888] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 6240.456182] Key type lgssc registered [ 6241.464935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6249.915174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6249.933235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6251.549075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6256.102360] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6261.734183] LustreError: 191247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6261.769074] LustreError: 191247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6265.880860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.374513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6271.100879] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6278.822511] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6284.511346] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6292.815831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6293.312301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6297.406213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6299.838268] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6307.293974] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6313.380293] Lustre: Failing over lustre-MDT0000 [ 6313.718643] Lustre: server umount lustre-MDT0000 complete [ 6315.058794] LustreError: 191244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.004022] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6316.016275] Lustre: Skipped 2 previous similar messages [ 6320.125840] LustreError: 192644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6320.160694] LustreError: 192644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6325.248487] LustreError: 191243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.271370] LustreError: 191243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6330.372291] LustreError: 191242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.394697] LustreError: 191242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6332.256172] Lustre: 190560:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790969451/real 1790969451] req@00000000ad5cb240 x1877967502336640/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790969467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6332.286358] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.491492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.609358] LustreError: 192252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.632219] LustreError: 192252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6341.648470] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcc9b506b445302c4 [ 6341.675298] Lustre: MGC192.168.204.111@tcp: Connection restored to 0@lo (at 0@lo) [ 6341.952372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.022679] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6346.751833] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6347.145545] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6347.253196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6347.280498] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6347.360609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6352.559463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6357.306614] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6363.107093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6363.129289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.145805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6367.201420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6367.209930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6367.220885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6369.337300] Lustre: server umount lustre-OST0000 complete [ 6377.960801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6377.961066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6377.993646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6378.006965] Lustre: Skipped 1 previous similar message [ 6378.018615] Lustre: Skipped 2 previous similar messages [ 6379.975396] Lustre: server umount lustre-MDT0000 complete [ 6383.081315] LustreError: 191248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.113222] LustreError: 191248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6384.830988] LustreError: 191227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969520 with bad export cookie 14743466226765464260 [ 6384.834055] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6384.849979] LustreError: 191227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6385.222818] Lustre: server umount lustre-MDT0001 complete [ 6398.604698] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6402.313889] Key type lgssc unregistered [ 6402.650148] LNet: 196044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6402.662589] LNetError: 196044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6402.691410] LNet: Removed LNI 192.168.204.111@tcp [ 6403.774365] Key type .llcrypt unregistered [ 6403.776952] Key type ._llcrypt unregistered [ 6429.449756] Key type ._llcrypt registered [ 6429.452179] Key type .llcrypt registered [ 6429.568374] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6431.169728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6431.190645] alg: No test for adler32 (adler32-zlib) [ 6432.190816] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 6432.334094] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 6433.977658] Key type lgssc registered [ 6435.399758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6444.911709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6444.938145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.400380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6451.084302] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6455.783742] LustreError: 197454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6455.816644] LustreError: 197454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6460.478507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.906648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6461.113098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6466.036849] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6473.545941] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6479.854422] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6487.178680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6487.540626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6490.630496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6493.643934] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6500.457587] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6504.359859] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6508.968501] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6527.818198] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6536.692937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.697137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6536.713943] Lustre: Skipped 1 previous similar message [ 6536.718329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6536.722432] LustreError: Skipped 1 previous similar message [ 6536.747362] Lustre: Skipped 1 previous similar message [ 6538.751449] Lustre: server umount lustre-OST0000 complete [ 6543.329698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6543.344945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.368919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6546.921152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6546.926084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6549.558076] Lustre: server umount lustre-MDT0000 complete [ 6552.033757] LustreError: 197454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6552.063042] LustreError: 197454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6554.353119] LustreError: 197434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969689 with bad export cookie 1274141276272791075 [ 6554.356370] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6554.360304] LustreError: 197434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6554.685308] Lustre: server umount lustre-MDT0001 complete [ 6569.210369] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6572.732915] Key type lgssc unregistered [ 6573.079223] LNet: 201320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6573.093487] LNetError: 201320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6573.128785] LNet: Removed LNI 192.168.204.111@tcp [ 6574.154514] Key type .llcrypt unregistered [ 6574.156952] Key type ._llcrypt unregistered [ 6595.282214] Key type ._llcrypt registered [ 6595.290209] Key type .llcrypt registered [ 6595.457955] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6596.470955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6596.490269] alg: No test for adler32 (adler32-zlib) [ 6597.567858] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 6597.762064] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 6599.386407] Key type lgssc registered [ 6600.831792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6610.403794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6610.440425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6612.019868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6616.670731] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6622.184384] LustreError: 202582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6622.203721] LustreError: 202582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6625.918345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6626.488131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6631.343613] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6639.101872] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6647.249107] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6656.968821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6657.457563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6658.493272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6665.402728] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6675.255778] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6682.592838] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6695.453322] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6702.707674] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6706.704849] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6716.900086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6716.901442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6716.939933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6722.025175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6722.031518] Lustre: Skipped 1 previous similar message [ 6727.139610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6727.144454] Lustre: Skipped 1 previous similar message [ 6731.232143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6731.380463] Lustre: server umount lustre-OST0000 complete [ 6737.379573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6737.395543] Lustre: Skipped 1 previous similar message [ 6737.404207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6737.409041] Lustre: Skipped 1 previous similar message [ 6741.966367] Lustre: server umount lustre-MDT0000 complete [ 6742.500343] LustreError: 202578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6742.533336] LustreError: 202578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6746.399608] LustreError: 205262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969881 with bad export cookie 16886769569593256683 [ 6746.406230] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6746.416244] LustreError: 205262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6746.897857] Lustre: server umount lustre-MDT0001 complete [ 6758.071132] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6761.116942] Key type lgssc unregistered [ 6761.482583] LNet: 207361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6761.492130] LNetError: 207361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6761.506620] LNet: Removed LNI 192.168.204.111@tcp [ 6762.673659] Key type .llcrypt unregistered [ 6762.675317] Key type ._llcrypt unregistered [ 6788.025616] Key type ._llcrypt registered [ 6788.032199] Key type .llcrypt registered [ 6788.217850] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6789.407198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6789.420787] alg: No test for adler32 (adler32-zlib) [ 6790.514766] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 6790.766362] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 6792.448195] Key type lgssc registered [ 6793.608419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6801.497348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6801.546504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.260608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.264311] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6812.656549] LustreError: 208775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.676860] LustreError: 208775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6817.772000] LustreError: 208774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6822.412610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6822.977844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6828.182869] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6836.857941] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6844.561670] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6853.566638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6854.196286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6862.676504] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6873.384329] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6873.663363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6893.037858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6893.046145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6893.076705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6894.050081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6894.051961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6894.068171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6897.994668] Lustre: server umount lustre-OST0000 complete [ 6904.291687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.305139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6904.332894] Lustre: Skipped 1 previous similar message [ 6904.338997] Lustre: Skipped 1 previous similar message [ 6908.421294] Lustre: server umount lustre-MDT0000 complete [ 6909.416612] LustreError: 210796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6909.455702] LustreError: 210796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6913.054240] LustreError: 208755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970048 with bad export cookie 1315715931012331681 [ 6913.076381] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6913.081764] LustreError: 208755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6913.749132] Lustre: server umount lustre-MDT0001 complete [ 6925.398067] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 6928.612216] Key type lgssc unregistered [ 6929.081938] LNet: 212457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6929.091033] LNetError: 212457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6929.110695] LNet: Removed LNI 192.168.204.111@tcp [ 6930.306838] Key type .llcrypt unregistered [ 6930.309380] Key type ._llcrypt unregistered [ 6950.713455] Key type ._llcrypt registered [ 6950.717828] Key type .llcrypt registered [ 6950.833743] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 6951.932524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6951.955800] alg: No test for adler32 (adler32-zlib) [ 6953.253726] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 6953.559334] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 6955.360209] Key type lgssc registered [ 6956.455741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6965.693744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6965.736845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6967.535513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6972.414529] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6978.021561] LustreError: 213719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6978.051467] LustreError: 213719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6982.535234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6982.935989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6988.439753] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 6997.551888] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7007.012135] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7014.397116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7014.967503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7020.811605] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7027.812756] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7037.545187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7052.778211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.787517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7052.803540] Lustre: Skipped 1 previous similar message [ 7052.814551] Lustre: Skipped 1 previous similar message [ 7057.893320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7058.367883] Lustre: server umount lustre-OST0000 complete [ 7063.010929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.019908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7063.021672] Lustre: Skipped 1 previous similar message [ 7063.031095] Lustre: Skipped 2 previous similar messages [ 7068.132610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7068.151694] Lustre: Skipped 1 previous similar message [ 7073.253165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7073.269444] Lustre: Skipped 1 previous similar message [ 7077.344586] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7077.771561] Lustre: server umount lustre-MDT0000 complete [ 7078.370006] LustreError: 214447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7078.396679] LustreError: 214447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7081.519350] LustreError: 213700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970217 with bad export cookie 101953992338707019 [ 7081.532415] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7081.541828] LustreError: 213700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7082.162403] Lustre: server umount lustre-MDT0001 complete [ 7093.350379] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 7096.697970] Key type lgssc unregistered [ 7097.094169] LNet: 217427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7097.104514] LNetError: 217427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7098.151605] LNet: Removed LNI 192.168.204.111@tcp [ 7099.344547] Key type .llcrypt unregistered [ 7099.347488] Key type ._llcrypt unregistered [ 7123.120603] Key type ._llcrypt registered [ 7123.125210] Key type .llcrypt registered [ 7123.379265] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 7124.967749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7125.287184] alg: No test for adler32 (adler32-zlib) [ 7126.313196] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 7126.702570] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 7128.560770] Key type lgssc registered [ 7130.448805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7140.713747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7140.748702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7142.195055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7146.245369] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7151.595503] LustreError: 218690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7151.629154] LustreError: 218690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7155.777099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.336072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7162.575623] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7172.734904] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7180.215382] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7188.246914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7188.909951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7195.305909] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7203.328199] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7212.458341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7223.035265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7223.121635] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7223.383897] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7231.467579] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7240.101350] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7242.163830] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7254.114288] Lustre: Permanently deactivating lustre-OST0001 [ 7254.158397] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7259.757809] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7265.682418] Lustre: Permanently reactivating lustre-OST0001 [ 7265.697283] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7265.704122] Lustre: Skipped 2 previous similar messages [ 7269.240829] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7269.270735] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7269.277110] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7269.290892] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7269.341738] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7277.489949] Lustre: server umount lustre-OST0001 complete [ 7277.565794] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7277.611631] Lustre: Skipped 1 previous similar message [ 7277.631151] LustreError: 220514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7277.685749] LustreError: 220514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7282.706118] LustreError: 221186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.730600] LustreError: 221186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7283.169820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7283.174995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.200017] Lustre: Skipped 1 previous similar message [ 7283.218948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7284.193819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7284.209551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7287.803260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7287.817770] Lustre: Skipped 1 previous similar message [ 7287.819189] LustreError: 220512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7287.864169] LustreError: 220512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7289.566932] Lustre: server umount lustre-OST0000 complete [ 7294.433026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7294.447340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.461560] Lustre: Skipped 1 previous similar message [ 7294.466796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7300.627474] Lustre: server umount lustre-MDT0000 complete [ 7303.138494] LustreError: 219940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7303.157901] LustreError: 219940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7304.909625] LustreError: 218672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970440 with bad export cookie 11039108337021426369 [ 7304.910702] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7304.918502] LustreError: 218672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7305.644699] Lustre: server umount lustre-MDT0001 complete [ 7318.995991] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 7322.237970] Key type lgssc unregistered [ 7322.664131] LNet: 223759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7322.675950] LNetError: 223759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7322.698382] LNet: Removed LNI 192.168.204.111@tcp [ 7323.659361] Key type .llcrypt unregistered [ 7323.662084] Key type ._llcrypt unregistered [ 7369.556827] Key type ._llcrypt registered [ 7369.559414] Key type .llcrypt registered [ 7369.649864] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 7371.161882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7371.209364] alg: No test for adler32 (adler32-zlib) [ 7372.300402] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 7372.596804] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 7374.280307] Key type lgssc registered [ 7375.436728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7383.922931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7383.939164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.212684] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7385.238752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7385.394658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7390.760918] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7401.784563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7401.887966] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7401.928742] Lustre: 225966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7402.352783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7407.709643] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7416.217847] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7422.558244] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7430.512737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7430.734330] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7430.920730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7436.426189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7437.998489] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7445.789818] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7449.704379] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7453.165937] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7462.696094] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7468.453637] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7477.919632] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7495.964773] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7495.971129] Lustre: Skipped 1 previous similar message [ 7533.459577] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7533.476797] Lustre: Skipped 3 previous similar messages [ 7604.340104] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7604.351616] Lustre: Skipped 8 previous similar messages [ 7630.821776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7630.828863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.851376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7631.333664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7631.350035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.367190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7631.371316] Lustre: Skipped 1 previous similar message [ 7636.135839] Lustre: server umount lustre-OST0000 complete [ 7641.575393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7641.579151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7641.604338] Lustre: Skipped 1 previous similar message [ 7641.621385] Lustre: Skipped 1 previous similar message [ 7646.690528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7646.699782] Lustre: Skipped 1 previous similar message [ 7646.792910] Lustre: server umount lustre-MDT0000 complete [ 7650.291555] LustreError: 229097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970785 with bad export cookie 3333498441635792540 [ 7650.294837] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7650.311877] LustreError: 229097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7650.626862] Lustre: server umount lustre-MDT0001 complete [ 7663.095926] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 7666.226825] Key type lgssc unregistered [ 7666.693302] LNet: 230764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7666.705532] LNetError: 230764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7666.732814] LNet: Removed LNI 192.168.204.111@tcp [ 7667.917211] Key type .llcrypt unregistered [ 7667.926582] Key type ._llcrypt unregistered [ 7694.976328] Key type ._llcrypt registered [ 7694.981457] Key type .llcrypt registered [ 7695.111758] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 7696.141059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7696.206528] alg: No test for adler32 (adler32-zlib) [ 7697.364313] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 7697.702904] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 7699.384219] Key type lgssc registered [ 7700.504472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7708.169839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7708.185975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7709.601668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7714.471366] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7719.911451] LustreError: 232178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7719.928382] LustreError: 232178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7723.541393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7724.077867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7729.450375] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7738.788826] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7745.312568] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7753.313627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7753.788950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7754.809141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7760.815720] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7770.335422] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7790.565852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7790.569798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7790.579174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7795.683774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7795.699578] Lustre: Skipped 2 previous similar messages [ 7795.830791] Lustre: server umount lustre-OST0000 complete [ 7800.807066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.810388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7800.810671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7800.834651] Lustre: Skipped 2 previous similar messages [ 7800.846524] Lustre: Skipped 1 previous similar message [ 7805.923779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7805.933839] Lustre: Skipped 1 previous similar message [ 7806.986272] Lustre: server umount lustre-MDT0000 complete [ 7811.047851] LustreError: 232905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7811.072238] LustreError: 232905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7811.781670] LustreError: 232158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970947 with bad export cookie 17092591099165684483 [ 7811.792252] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7811.808070] LustreError: 232158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7812.241552] Lustre: server umount lustre-MDT0001 complete [ 7824.847295] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 7828.015911] Key type lgssc unregistered [ 7828.371786] LNet: 235876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7828.386849] LNetError: 235876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7828.401795] LNet: Removed LNI 192.168.204.111@tcp [ 7829.648188] Key type .llcrypt unregistered [ 7829.659732] Key type ._llcrypt unregistered [ 7856.604249] Key type ._llcrypt registered [ 7856.611613] Key type .llcrypt registered [ 7856.772826] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 7858.218271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7858.238880] alg: No test for adler32 (adler32-zlib) [ 7859.396796] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 7859.716256] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 7861.488793] Key type lgssc registered [ 7862.937520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8136.171963] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 8139.006352] Key type lgssc unregistered [ 8139.296401] LNet: 237921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8139.312261] LNetError: 237921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8139.328862] LNet: Removed LNI 192.168.204.111@tcp [ 8140.057714] Key type .llcrypt unregistered [ 8140.063041] Key type ._llcrypt unregistered [ 8168.765659] Key type ._llcrypt registered [ 8168.771402] Key type .llcrypt registered [ 8168.909956] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 8189.818586] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8191.367287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8191.384863] alg: No test for adler32 (adler32-zlib) [ 8192.540607] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 8192.808805] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 8194.520226] Key type lgssc registered [ 8195.883964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8230.212623] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7865 sec ========== 16:02:44 (1790971364) [ 8232.092922] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:02:46 (1790971366) === [ 8236.355470] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:02:50 (1790971370) === [ 8286.076535] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 8289.229245] Key type lgssc unregistered [ 8289.524798] LNet: 243159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8289.534057] LNetError: 243159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8289.555719] LNet: Removed LNI 192.168.204.111@tcp [ 8290.648568] Key type .llcrypt unregistered [ 8290.652149] Key type ._llcrypt unregistered