[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 668918734 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002012] Switched APIC routing to physical x2apic. [ 0.003012] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007011] pid_max: default: 32768 minimum: 301 [ 0.008145] LSM: Security Framework initializing [ 0.009047] Yama: becoming mindful. [ 0.010028] SELinux: Initializing. [ 0.011059] *** VALIDATE selinux *** [ 0.019341] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024049] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026128] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] *** VALIDATE tmpfs *** [ 0.029479] *** VALIDATE proc *** [ 0.031160] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.034006] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041012] debug: unmapping init [mem 0xffffffff94459000-0xffffffff94460fff] [ 0.043793] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044597] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.050869] ... event mask: 000000070000000f [ 0.051246] rcu: Hierarchical SRCU implementation. [ 0.053277] smp: Bringing up secondary CPUs ... [ 0.054515] x86: Booting SMP configuration: [ 0.055023] .... node #0, CPUs: #1 #2 #3 [ 0.060111] smp: Brought up 1 node, 4 CPUs [ 0.062010] smpboot: Max logical packages: 1 [ 0.063012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.106173] node 0 deferred pages initialised in 41ms [ 0.109136] devtmpfs: initialized [ 0.110122] x86/mm: Memory block size: 128MB [ 0.114037] gcov: version magic: 0x41383552 [ 0.117519] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119130] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.122242] pinctrl core: initialized pinctrl subsystem [ 0.124165] [ 0.124884] ************************************************************* [ 0.126010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127007] ** ** [ 0.129022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.131009] ** ** [ 0.133012] ** This means that this kernel is built to expose internal ** [ 0.134050] ** IOMMU data structures, which may compromise security on ** [ 0.136045] ** your system. ** [ 0.137006] ** ** [ 0.138038] ** If you see this message and you are not debugging the ** [ 0.140006] ** kernel, report this immediately to your vendor! ** [ 0.141005] ** ** [ 0.142034] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143009] ************************************************************* [ 0.146460] NET: Registered protocol family 16 [ 0.148411] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.151046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.153065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.156100] cpuidle: using governor menu [ 0.157937] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.159428] PCI: Using configuration type 1 for base access [ 0.161112] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.168245] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.171013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.173170] cryptd: max_cpu_qlen set to 1000 [ 0.175215] ACPI: Added _OSI(Module Device) [ 0.177021] ACPI: Added _OSI(Processor Device) [ 0.178010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.180023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.186205] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.193348] ACPI: Interpreter enabled [ 0.195058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.197013] ACPI: Using IOAPIC for interrupt routing [ 0.199166] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202567] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.214000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.215051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.218017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.225760] acpiphp: Slot [2] registered [ 0.227117] acpiphp: Slot [5] registered [ 0.229126] acpiphp: Slot [6] registered [ 0.230108] acpiphp: Slot [7] registered [ 0.231090] acpiphp: Slot [8] registered [ 0.232061] acpiphp: Slot [9] registered [ 0.233084] acpiphp: Slot [10] registered [ 0.234102] acpiphp: Slot [3] registered [ 0.235136] acpiphp: Slot [4] registered [ 0.237105] acpiphp: Slot [11] registered [ 0.238102] acpiphp: Slot [12] registered [ 0.240097] acpiphp: Slot [13] registered [ 0.241121] acpiphp: Slot [14] registered [ 0.242094] acpiphp: Slot [15] registered [ 0.243059] acpiphp: Slot [16] registered [ 0.244163] acpiphp: Slot [17] registered [ 0.246090] acpiphp: Slot [18] registered [ 0.247073] acpiphp: Slot [19] registered [ 0.248124] acpiphp: Slot [20] registered [ 0.249086] acpiphp: Slot [21] registered [ 0.250082] acpiphp: Slot [22] registered [ 0.251080] acpiphp: Slot [23] registered [ 0.252100] acpiphp: Slot [24] registered [ 0.253023] acpiphp: Slot [25] registered [ 0.254047] acpiphp: Slot [26] registered [ 0.255126] acpiphp: Slot [27] registered [ 0.256191] acpiphp: Slot [28] registered [ 0.257100] acpiphp: Slot [29] registered [ 0.258105] acpiphp: Slot [30] registered [ 0.259000] acpiphp: Slot [31] registered [ 0.259000] PCI host bridge to bus 0000:00 [ 0.260019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.265037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.267038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.270028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.272017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.274013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.275169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.277704] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.281124] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.290000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.295049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.297017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.298011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.301017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.303450] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.307915] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.310040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.312916] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.319016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.332014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.337014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.344040] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.351020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.358020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.373015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.385594] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.393018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.400020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.422020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.460726] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.467013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.471021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.488033] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.505237] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.520018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.531025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.549022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.566994] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.573017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.580016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.603021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.619983] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.632021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.643022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.672021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.685764] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.687353] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.690258] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.691323] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.694493] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.701000] iommu: Default domain type: Passthrough [ 0.702365] SCSI subsystem initialized [ 0.704023] ACPI: bus type USB registered [ 0.707066] usbcore: registered new interface driver usbfs [ 0.710096] usbcore: registered new interface driver hub [ 0.712100] usbcore: registered new device driver usb [ 0.714191] pps_core: LinuxPPS API ver. 1 registered [ 0.716012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.720068] PTP clock support registered [ 0.722061] EDAC MC: Ver: 3.0.0 [ 0.724184] PCI: Using ACPI for IRQ routing [ 0.726828] NetLabel: Initializing [ 0.728015] NetLabel: domain hash size = 128 [ 0.730015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.732089] NetLabel: unlabeled traffic allowed by default [ 0.735244] vgaarb: loaded [ 0.736328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.738014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.745325] clocksource: Switched to clocksource kvm-clock [ 0.867535] VFS: Disk quotas dquot_6.6.0 [ 0.868566] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.870058] *** VALIDATE ramfs *** [ 0.870899] *** VALIDATE hugetlbfs *** [ 0.871977] pnp: PnP ACPI init [ 0.874292] pnp: PnP ACPI: found 6 devices [ 0.890810] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.893826] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.895864] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.897661] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.899650] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.901795] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.904667] NET: Registered protocol family 2 [ 0.907556] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.913960] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.919538] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.926587] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.930690] TCP: Hash tables configured (established 65536 bind 65536) [ 0.934270] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.938793] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.942550] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.945629] NET: Registered protocol family 1 [ 0.948653] RPC: Registered named UNIX socket transport module. [ 0.950838] RPC: Registered udp transport module. [ 0.952767] RPC: Registered tcp transport module. [ 0.954464] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.956739] NET: Registered protocol family 44 [ 0.958125] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.959944] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.961693] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.963655] PCI: CLS 0 bytes, default 64 [ 0.964987] Unpacking initramfs... [ 2.486198] debug: unmapping init [mem 0xffff8c857cc54000-0xffff8c857ffbffff] [ 2.490072] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.491865] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.494277] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.050796] Initialise system trusted keyrings [ 3.052389] Key type blacklist registered [ 3.054252] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.063205] zbud: loaded [ 3.066812] *** VALIDATE nfs *** [ 3.068121] *** VALIDATE nfs4 *** [ 3.070672] pstore: using deflate compression [ 3.075044] Platform Keyring initialized [ 3.225866] NET: Registered protocol family 38 [ 3.227593] Key type asymmetric registered [ 3.229235] Asymmetric key parser 'x509' registered [ 3.231480] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.235028] io scheduler mq-deadline registered [ 3.236571] io scheduler kyber registered [ 3.238849] io scheduler bfq registered [ 3.241428] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.245289] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.249875] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.253693] ACPI: Power Button [PWRF] [ 3.260784] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.270554] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.295226] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.308518] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.340941] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.370086] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.400440] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.405879] Non-volatile memory driver v1.3 [ 3.407717] Linux agpgart interface v0.103 [ 3.446036] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.449093] vda: detected capacity change from 0 to 74690560 [ 3.473861] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.477732] vdb: detected capacity change from 0 to 1073741824 [ 3.500258] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.503826] vdc: detected capacity change from 0 to 2621440000 [ 3.531463] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.535253] vdd: detected capacity change from 0 to 2621440000 [ 3.560803] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.564555] vde: detected capacity change from 0 to 4294967296 [ 3.592872] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.596720] vdf: detected capacity change from 0 to 4294967296 [ 3.610378] libphy: Fixed MDIO Bus: probed [ 3.633218] usbcore: registered new interface driver usbserial_generic [ 3.635967] usbserial: USB Serial support registered for generic [ 3.638783] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.643962] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.646266] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.649454] mousedev: PS/2 mouse device common for all mice [ 3.654662] rtc_cmos 00:05: RTC can wake from S4 [ 3.656299] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.660127] rtc_cmos 00:05: registered as rtc0 [ 3.664193] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.709927] hpet1: lost 2 rtc interrupts [ 3.713252] intel_pstate: CPU model not supported [ 3.716384] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.721091] hid: raw HID events driver (C) Jiri Kosina [ 3.728172] usbcore: registered new interface driver usbhid [ 3.737682] usbhid: USB HID core driver [ 3.740918] drop_monitor: Initializing network drop monitor service [ 3.745313] Initializing XFRM netlink socket [ 3.747182] NET: Registered protocol family 10 [ 3.748636] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.752456] Segment Routing with IPv6 [ 3.757321] NET: Registered protocol family 17 [ 3.759563] mpls_gso: MPLS GSO support [ 3.765885] RAS: Correctable Errors collector initialized. [ 3.768233] AVX version of gcm_enc/dec engaged. [ 3.769989] AES CTR mode by8 optimization enabled [ 3.876975] sched_clock: Marking stable (3876887227, 0)->(4982282062, -1105394835) [ 3.884052] registered taskstats version 1 [ 3.886333] Loading compiled-in X.509 certificates [ 3.889340] zswap: loaded using pool lzo/zbud [ 3.939488] Key type big_key registered [ 3.959142] Key type encrypted registered [ 3.960650] ima: No TPM chip found, activating TPM-bypass! [ 3.962246] ima: Allocated hash algorithm: sha1 [ 3.963675] ima: No architecture policies found [ 3.965059] evm: Initialising EVM extended attributes: [ 3.966958] evm: security.selinux [ 3.968679] evm: security.ima [ 3.970204] evm: security.capability [ 3.971935] evm: HMAC attrs: 0x1 [ 3.974846] rtc_cmos 00:05: setting system clock to 2026-09-02 14:49:53 UTC (1788360593) [ 3.982643] debug: unmapping init [mem 0xffffffff95403000-0xffffffff955fffff] [ 3.987246] debug: unmapping init [mem 0xffffffff94182000-0xffffffff94458fff] [ 3.994129] Write protecting the kernel read-only data: 28672k [ 4.000614] debug: unmapping init [mem 0xffffffff92803000-0xffffffff929fffff] [ 4.004514] debug: unmapping init [mem 0xffffffff93114000-0xffffffff931fffff] [ 4.054596] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.066662] systemd[1]: Detected virtualization kvm. [ 4.069611] systemd[1]: Detected architecture x86-64. [ 4.072179] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.102764] systemd[1]: No hostname configured. [ 4.104306] systemd[1]: Set hostname to . [ 4.106557] random: systemd: uninitialized urandom read (16 bytes read) [ 4.108572] systemd[1]: Initializing machine ID from random generator. [ 4.346446] random: systemd: uninitialized urandom read (16 bytes read) [ 4.348724] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.355421] random: systemd: uninitialized urandom read (16 bytes read) [ 4.359508] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.368158] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.540851] device-mapper: uevent: version 1.0.3 [ 7.544315] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 8.691053] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 8.765121] random: fast init done [ 9.495551] scsi host0: ata_piix [ 9.595574] scsi host1: ata_piix [ 9.597988] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.608811] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.670431] random: crng init done [ 15.677518] random: 7 urandom warning(s) missed due to ratelimiting [ 18.491660] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.406343] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.329394] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.745187] SELinux: Disabled at runtime. [ 23.814280] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.825232] systemd[1]: Detected virtualization kvm. [ 23.827410] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.912950] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.917270] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.928752] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.933577] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.938804] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.948751] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.964776] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ 25.548563] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt[ 26.647431] squashfs: version 4.0 (2009/01/31) Phillip Lougher . [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.793232] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.822638] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.637500] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.787696] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 34.085782] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (9s / no limit)[ 34.490811] NFS: Registering the id_resolver key type [ 34.493421] Key type id_resolver registered [ 34.496275] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg156-server login: [ 117.234647] libcfs: loading out-of-tree module taints kernel. [ 117.267760] Key type ._llcrypt registered [ 117.271547] Key type .llcrypt registered [ 117.374399] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 134.553526] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 136.321615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 136.345781] alg: No test for adler32 (adler32-zlib) [ 137.826350] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 138.505222] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 140.200186] Key type lgssc registered [ 141.762362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 158.743431] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 176.572158] hrtimer: interrupt took 2365884 ns [ 196.783819] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 208.986309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 209.027648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.362491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 210.414569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 210.528739] Lustre: lustre-MDT0000: new disk, initializing [ 210.626070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.653180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 214.753361] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.587925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.678321] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 226.705344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 226.709248] Lustre: Skipped 1 previous similar message [ 226.769967] Lustre: lustre-MDT0001: new disk, initializing [ 226.823294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.843276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 226.864337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 230.994232] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.891920] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.034636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.292655] Lustre: lustre-OST0000: new disk, initializing [ 244.297797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 244.309225] Lustre: 8459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.412719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.362863] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.928067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 251.947717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 252.037138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 264.259698] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.389365] Lustre: lustre-OST0001: new disk, initializing [ 264.392847] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.398114] Lustre: 9532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.477870] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 270.947909] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.454237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 273.463152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 273.553377] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 282.638597] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 292.021532] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 298.184453] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing check_logdir /tmp/testlogs/ [ 303.226601] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing yml_node [ 308.323261] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 310.861712] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 312.768919] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 314.139537] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 2 10:55:02 EDT 2026 [ 330.144181] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 331.594967] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 349.664832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 349.672197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.684192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.181543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.191900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 350.197719] Lustre: Skipped 2 previous similar messages [ 354.051970] Lustre: server umount lustre-MDT0000 complete [ 357.528373] LustreError: 10347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360947 with bad export cookie 5947814407590648834 [ 357.536774] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.540372] LustreError: 10347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 360.416975] LustreError: 6528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.417189] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.422079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.430813] LustreError: 6528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 360.440637] Lustre: Skipped 3 previous similar messages [ 362.465045] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.465768] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 362.496304] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 362.519802] Lustre: Skipped 2 previous similar messages [ 363.908923] Lustre: server umount lustre-MDT0001 complete [ 367.577296] Lustre: server umount lustre-OST0000 complete [ 370.671868] Lustre: server umount lustre-OST0001 complete [ 376.669236] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 384.825587] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 413.158034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.298406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.316336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 413.394307] Lustre: lustre-MDT0000: new disk, initializing [ 413.440159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.450667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.655865] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 425.984965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.092525] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 426.138935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 426.142544] Lustre: Skipped 1 previous similar message [ 426.194388] Lustre: lustre-MDT0001: new disk, initializing [ 426.273016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 426.304283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 426.316499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 429.775699] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 438.111144] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 442.858923] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 449.399670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 449.716987] Lustre: lustre-OST0000: new disk, initializing [ 449.725741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 449.737472] Lustre: 17048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 449.847971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.470221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 451.477564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 451.600176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 455.214414] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 463.367308] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 466.842386] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.033302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 470.481559] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 470.605845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 472.036605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.044667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 472.049605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 472.055967] LustreError: Skipped 1 previous similar message [ 472.061922] Lustre: Skipped 1 previous similar message [ 482.284827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 482.291944] Lustre: Skipped 2 previous similar messages [ 486.368775] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 486.531812] Lustre: server umount lustre-OST0000 complete [ 492.513722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.523329] Lustre: Skipped 1 previous similar message [ 495.540729] Lustre: server umount lustre-MDT0000 complete [ 497.639453] LustreError: 15091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.649059] LustreError: 15091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 498.148693] LustreError: 15884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361087 with bad export cookie 5947814407590650367 [ 498.151226] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 498.157250] LustreError: 15884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 498.385952] Lustre: server umount lustre-MDT0001 complete [ 500.795757] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 10:58:09 (1788361089) [ 505.719296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.011205] LustreError: 19314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.063347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.152199] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 511.460792] LustreError: 19315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.881737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.576607] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 526.276412] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 530.784488] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 535.578673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 539.853610] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 544.876828] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 561.642782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.643290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 561.655419] Lustre: Skipped 1 previous similar message [ 561.664259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 561.673273] Lustre: Skipped 3 previous similar messages [ 566.769881] Lustre: server umount lustre-OST0000 complete [ 576.410808] Lustre: server umount lustre-MDT0000 complete [ 577.004163] LustreError: 19315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.020166] LustreError: 19315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 579.797546] LustreError: 19295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361169 with bad export cookie 5947814407590651249 [ 579.800762] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.807613] LustreError: 19295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 580.164245] Lustre: server umount lustre-MDT0001 complete [ 591.094925] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 594.058855] Key type lgssc unregistered [ 594.395598] LNet: 22868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.402862] LNetError: 22868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.414263] LNet: Removed LNI 192.168.201.156@tcp [ 595.424189] Key type .llcrypt unregistered [ 595.429102] Key type ._llcrypt unregistered [ 614.115856] Key type ._llcrypt registered [ 614.117789] Key type .llcrypt registered [ 614.201366] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 614.844325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 614.868545] alg: No test for adler32 (adler32-zlib) [ 615.951436] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 616.156965] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 617.816358] Key type lgssc registered [ 618.730837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 624.801274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 624.816461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.138094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 629.575438] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 636.386760] LustreError: 24260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.399074] LustreError: 24260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 636.582547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.978336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 640.697086] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 647.388354] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 651.989137] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 657.327307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 657.616786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.622387] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 668.307488] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 670.964499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 681.444520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.445909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 681.458331] Lustre: Skipped 1 previous similar message [ 681.467530] Lustre: Skipped 1 previous similar message [ 683.331389] Lustre: server umount lustre-OST0000 complete [ 688.097770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 688.106536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 688.116391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 691.681366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 691.694240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.802096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 696.807862] Lustre: Skipped 2 previous similar messages [ 701.408166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 701.585307] Lustre: server umount lustre-MDT0000 complete [ 701.928973] LustreError: 25323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.949154] LustreError: 25323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 704.482320] LustreError: 24241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361294 with bad export cookie 12456779581933483852 [ 704.487347] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.494416] LustreError: 24241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 704.761564] Lustre: server umount lustre-MDT0001 complete [ 713.535259] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 715.405389] Key type lgssc unregistered [ 715.634042] LNet: 28246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.651348] LNetError: 28246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 715.673673] LNet: Removed LNI 192.168.201.156@tcp [ 716.319882] Key type .llcrypt unregistered [ 716.322368] Key type ._llcrypt unregistered [ 734.633789] Key type ._llcrypt registered [ 734.637622] Key type .llcrypt registered [ 734.702640] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 735.483641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 735.586386] alg: No test for adler32 (adler32-zlib) [ 736.563729] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 736.799432] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 738.433399] Key type lgssc registered [ 739.143183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 745.051328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 745.063902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.420842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 750.096823] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 755.684307] LustreError: 29637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 755.699566] LustreError: 29637:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 757.160108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 757.586409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 761.926653] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 768.413970] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 773.312453] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 786.632578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 787.027310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.172124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 793.110450] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 798.959181] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 812.002306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 812.007844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.016811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 814.057398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 814.059701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 814.066409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 815.842952] Lustre: server umount lustre-OST0000 complete [ 819.170036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.173412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 819.183520] Lustre: Skipped 1 previous similar message [ 819.193829] Lustre: Skipped 2 previous similar messages [ 824.290368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 824.294708] Lustre: Skipped 1 previous similar message [ 825.119794] Lustre: server umount lustre-MDT0000 complete [ 828.097126] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361417 with bad export cookie 17933392929056386597 [ 828.105406] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 828.109030] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 828.328155] Lustre: server umount lustre-MDT0001 complete [ 836.608297] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 838.691972] Key type lgssc unregistered [ 838.923723] LNet: 33515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.928265] LNetError: 33515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.944173] LNet: Removed LNI 192.168.201.156@tcp [ 839.589342] Key type .llcrypt unregistered [ 839.593900] Key type ._llcrypt unregistered [ 856.553597] Key type ._llcrypt registered [ 856.554875] Key type .llcrypt registered [ 856.628945] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 857.376726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 857.447737] alg: No test for adler32 (adler32-zlib) [ 858.386677] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 858.542551] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 860.160174] Key type lgssc registered [ 860.975214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 866.495774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 866.512605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.797205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 871.090440] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 877.826953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 878.116129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 881.737753] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 887.598849] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 891.828081] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 896.913938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 897.189211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.299157] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 907.716890] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 913.591080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 926.181733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 926.191722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.206655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 929.251737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 929.253274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.256104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 930.962763] Lustre: server umount lustre-OST0000 complete [ 934.372978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 934.373592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.383290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 934.388255] Lustre: Skipped 1 previous similar message [ 934.393664] Lustre: Skipped 2 previous similar messages [ 939.490327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 939.493866] Lustre: Skipped 1 previous similar message [ 944.613834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 944.618678] Lustre: Skipped 1 previous similar message [ 948.192146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 948.331176] Lustre: server umount lustre-MDT0000 complete [ 948.712484] LustreError: 34904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.731318] LustreError: 34904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 951.088482] LustreError: 34885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361540 with bad export cookie 3349156862837782096 [ 951.093919] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 951.094396] LustreError: 34885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 951.320922] Lustre: server umount lustre-MDT0001 complete [ 959.064516] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 961.002684] Key type lgssc unregistered [ 961.252669] LNet: 38462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.256507] LNetError: 38462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.275287] LNet: Removed LNI 192.168.201.156@tcp [ 961.909327] Key type .llcrypt unregistered [ 961.911437] Key type ._llcrypt unregistered [ 976.821563] Key type ._llcrypt registered [ 976.823530] Key type .llcrypt registered [ 976.899766] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 977.593824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 977.669380] alg: No test for adler32 (adler32-zlib) [ 978.644262] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 978.809135] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 980.440178] Key type lgssc registered [ 981.145326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 985.937776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 985.952694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.227767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.254608] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 995.952832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.222431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 999.171903] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1004.242983] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1008.097223] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1013.179666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1013.438785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.006807] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1023.437371] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1031.863256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1040.605589] Lustre: lustre-OST0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 1041.381415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1041.392631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.412889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1042.406210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1042.407430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.432675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1042.439344] Lustre: Skipped 1 previous similar message [ 1045.180880] Lustre: server umount lustre-OST0000 complete [ 1061.908315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.917473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.069338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1068.075297] Lustre: Skipped 4 previous similar messages [ 1068.206495] Lustre: server umount lustre-MDT0000 complete [ 1071.158538] LustreError: 39837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361660 with bad export cookie 13590450362684528023 [ 1071.164853] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1077.532371] Lustre: server umount lustre-MDT0001 complete [ 1085.451700] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1087.351504] Key type lgssc unregistered [ 1087.629600] LNet: 43582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.638315] LNetError: 43582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.660263] LNet: Removed LNI 192.168.201.156@tcp [ 1088.398377] Key type .llcrypt unregistered [ 1088.401115] Key type ._llcrypt unregistered [ 1105.783189] Key type ._llcrypt registered [ 1105.785619] Key type .llcrypt registered [ 1105.862060] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1106.486503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1106.500685] alg: No test for adler32 (adler32-zlib) [ 1107.495565] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1107.678889] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1109.304312] Key type lgssc registered [ 1110.241510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1115.597207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1115.619087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.909531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.050416] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1126.311279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.558918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1128.962194] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1134.114963] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1138.126792] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1142.587052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1142.881850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.916880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1146.719924] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1151.475697] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1157.604881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1157.605563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.612157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1157.619388] Lustre: Skipped 2 previous similar messages [ 1157.624349] Lustre: Skipped 1 previous similar message [ 1159.396879] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 1159.405849] Lustre: Skipped 1 previous similar message [ 1159.755203] Lustre: server umount lustre-MDT0000 complete [ 1162.050235] LustreError: 44954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361751 with bad export cookie 13563988638132432012 [ 1162.054211] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1162.057203] LustreError: 44954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1162.723196] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.725620] LustreError: 44969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.741074] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1162.747754] LustreError: 44969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1164.501494] LustreError: 44968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.841885] LustreError: 44967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.841966] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1167.865117] Lustre: Skipped 1 previous similar message [ 1168.410344] Lustre: server umount lustre-MDT0001 complete [ 1180.382974] LustreError: 44952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.56@tcp arrived at 1788361769 with bad export cookie 13563988638132432901 [ 1180.395088] LustreError: 44952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1188.555908] Lustre: server umount lustre-OST0000 complete [ 1199.933630] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1201.893897] Key type lgssc unregistered [ 1202.118803] LNet: 48828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.124873] LNetError: 48828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.175866] LNet: Removed LNI 192.168.201.156@tcp [ 1203.755870] Key type .llcrypt unregistered [ 1203.757816] Key type ._llcrypt unregistered [ 1220.996537] Key type ._llcrypt registered [ 1220.997802] Key type .llcrypt registered [ 1221.085578] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1221.603833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1221.625608] alg: No test for adler32 (adler32-zlib) [ 1222.640355] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1222.806019] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1224.440832] Key type lgssc registered [ 1225.379171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1231.001202] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1231.047695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1246.560436] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1251.680320] LustreError: 50195:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 1278.304492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.404855] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1287.315305] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1452.161955] Lustre: server umount lustre-OST0000 complete [ 1465.158932] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1467.204275] Key type lgssc unregistered [ 1467.484178] LNet: 51870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.495421] LNetError: 51870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.513686] LNet: Removed LNI 192.168.201.156@tcp [ 1468.092360] Key type .llcrypt unregistered [ 1468.093893] Key type ._llcrypt unregistered [ 1484.056179] Key type ._llcrypt registered [ 1484.058415] Key type .llcrypt registered [ 1484.133472] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1484.984264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1484.996875] alg: No test for adler32 (adler32-zlib) [ 1486.013803] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1486.191571] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1487.843895] Key type lgssc registered [ 1488.631266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1494.268694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1494.281660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1495.583978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.733680] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1505.177749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.462300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1508.673657] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1514.286583] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1519.479352] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1524.388940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1524.706133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1530.137156] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1535.245788] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1540.068468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.097094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1541.160136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1542.849343] Lustre: server umount lustre-OST0000 complete [ 1546.217110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1546.238937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1546.250347] Lustre: Skipped 2 previous similar messages [ 1551.330384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1551.340130] Lustre: Skipped 1 previous similar message [ 1551.883708] Lustre: server umount lustre-MDT0000 complete [ 1554.223712] LustreError: 53258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362143 with bad export cookie 3165577248962127489 [ 1554.232385] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1554.236167] LustreError: 53258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1554.399124] Lustre: server umount lustre-MDT0001 complete [ 1561.339588] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1563.222926] Key type lgssc unregistered [ 1563.431897] LNet: 56805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.436256] LNetError: 56805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.454388] LNet: Removed LNI 192.168.201.156@tcp [ 1563.980599] Key type .llcrypt unregistered [ 1563.983218] Key type ._llcrypt unregistered [ 1578.422094] Key type ._llcrypt registered [ 1578.423883] Key type .llcrypt registered [ 1578.480882] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1579.065632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1579.121357] alg: No test for adler32 (adler32-zlib) [ 1580.079239] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1580.282112] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1581.920219] Key type lgssc registered [ 1582.687723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1587.874561] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1587.888732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1603.362240] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1608.481190] LustreError: 58170:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 1635.168207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.365449] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1643.806448] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1647.927510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.228736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.281772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1651.126207] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1655.790184] LustreError: 58195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1655.805141] LustreError: 58195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1656.284813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1656.509299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1659.307683] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1664.383246] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1668.724301] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1672.168700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1672.172645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.177543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1672.189882] Lustre: Skipped 1 previous similar message [ 1676.622960] Lustre: server umount lustre-OST0000 complete [ 1696.737459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.754224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1696.759443] Lustre: Skipped 2 previous similar messages [ 1701.859126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.863691] Lustre: Skipped 1 previous similar message [ 1706.978363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1706.991978] Lustre: Skipped 1 previous similar message [ 1710.052794] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1710.167963] Lustre: server umount lustre-MDT0000 complete [ 1712.101343] LustreError: 60897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1712.111366] LustreError: 60897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1712.632814] LustreError: 58177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362302 with bad export cookie 9947678904342355758 [ 1712.642555] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1712.644053] LustreError: 58177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1718.943344] Lustre: server umount lustre-MDT0001 complete [ 1726.356630] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1728.280208] Key type lgssc unregistered [ 1728.522737] LNet: 61912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.532318] LNetError: 61912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.546185] LNet: Removed LNI 192.168.201.156@tcp [ 1729.141229] Key type .llcrypt unregistered [ 1729.142592] Key type ._llcrypt unregistered [ 1745.986580] Key type ._llcrypt registered [ 1745.988687] Key type .llcrypt registered [ 1746.066328] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1746.877391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.997408] alg: No test for adler32 (adler32-zlib) [ 1748.000613] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1748.169717] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1749.808185] Key type lgssc registered [ 1750.556238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1755.492649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1755.506072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.801792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.617087] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1764.827208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.066897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1768.169604] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1772.638625] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1775.725800] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1780.048743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1780.258480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.326453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1784.068834] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1788.280844] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1810.917086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.920756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1810.934487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1815.463576] Lustre: server umount lustre-OST0000 complete [ 1821.154028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.164643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.166224] Lustre: Skipped 2 previous similar messages [ 1821.174602] Lustre: Skipped 2 previous similar messages [ 1824.313729] Lustre: server umount lustre-MDT0000 complete [ 1826.273745] LustreError: 63300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.293560] LustreError: 63300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1827.284505] LustreError: 63280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362416 with bad export cookie 10692809702441346113 [ 1827.288332] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.294574] LustreError: 63280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1827.571796] Lustre: server umount lustre-MDT0001 complete [ 1835.418817] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1837.311525] Key type lgssc unregistered [ 1837.525279] LNet: 66851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.531705] LNetError: 66851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.550215] LNet: Removed LNI 192.168.201.156@tcp [ 1838.188241] Key type .llcrypt unregistered [ 1838.190146] Key type ._llcrypt unregistered [ 1854.666240] Key type ._llcrypt registered [ 1854.668190] Key type .llcrypt registered [ 1854.723537] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1855.450744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1855.466882] alg: No test for adler32 (adler32-zlib) [ 1856.403028] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1856.573850] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1858.200336] Key type lgssc registered [ 1858.966378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1864.515983] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 11:20:53 (1788362453) [ 1868.816520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1868.832396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.091381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.579243] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1878.105434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.310366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1881.081512] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1885.816674] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1889.462991] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1893.945512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1894.164344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.230444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1897.962820] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1902.470401] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1907.502218] Lustre: Failing over lustre-MDT0000 [ 1907.656844] Lustre: server umount lustre-MDT0000 complete [ 1908.715258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.719641] LustreError: 69452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.725730] Lustre: Skipped 1 previous similar message [ 1908.744429] LustreError: 69452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1909.976605] LustreError: 68532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.588939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1913.677698] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1913.765770] Lustre: *** cfs_fail_loc=135, val=0*** [ 1913.828105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1913.842764] LustreError: 71619:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1913.849914] LustreError: 71619:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.156@tcp: cfg command failed: rc = -2 [ 1913.853460] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1913.853460] [ 1913.859955] LustreError: MGC192.168.201.156@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1913.868703] LustreError: 71607:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1913.875056] LustreError: 71607:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 1913.878570] LustreError: 71607:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 1913.887150] LustreError: 71607:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1913.916689] Lustre: server umount lustre-MDT0000 complete [ 1913.920608] LustreError: 71607:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1915.103601] LustreError: 68531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.687716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.785463] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1918.961685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1919.019087] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1920.206052] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1921.703152] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1924.080362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1924.114489] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1924.144307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1929.185646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1929.195044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.199979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1929.203476] Lustre: Skipped 2 previous similar messages [ 1929.209015] Lustre: Skipped 2 previous similar messages [ 1932.430438] Lustre: server umount lustre-OST0000 complete [ 1939.426324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.426992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1939.437832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.450702] Lustre: Skipped 2 previous similar messages [ 1940.801203] Lustre: server umount lustre-MDT0000 complete [ 1942.996403] LustreError: 68516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362532 with bad export cookie 16321075550692793861 [ 1943.004338] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1943.006488] LustreError: 68516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1943.270696] Lustre: server umount lustre-MDT0001 complete [ 1949.274935] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1950.770837] Key type lgssc unregistered [ 1950.949779] LNet: 73394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.954839] LNetError: 73394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1950.968799] LNet: Removed LNI 192.168.201.156@tcp [ 1951.496489] Key type .llcrypt unregistered [ 1951.498106] Key type ._llcrypt unregistered [ 1964.903547] Key type ._llcrypt registered [ 1964.905955] Key type .llcrypt registered [ 1965.013845] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1965.652066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1965.813582] alg: No test for adler32 (adler32-zlib) [ 1966.767564] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1966.937056] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1968.560179] Key type lgssc registered [ 1969.240397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1973.708809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1973.718301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.928281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.043148] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1981.990839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.210426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1984.426066] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1989.128809] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1992.855285] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1998.392673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1998.629615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.658351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2002.040710] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2006.228873] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2013.847511] Lustre: Failing over lustre-MDT0000 [ 2014.058947] Lustre: server umount lustre-MDT0000 complete [ 2018.003514] LustreError: 76641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.014971] LustreError: 76641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2018.277368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2018.278347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.292960] Lustre: Skipped 2 previous similar messages [ 2019.243734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.338332] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.441268] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2019.543950] LustreError: 77884:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2019.547641] LustreError: 77884:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.156@tcp: cfg command failed: rc = -9 [ 2019.552189] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2019.552189] [ 2019.560113] LustreError: MGC192.168.201.156@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2019.569153] LustreError: 77873:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2019.573795] LustreError: 77873:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2019.577612] LustreError: 77873:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2019.582260] LustreError: 77873:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2019.597924] Lustre: server umount lustre-MDT0000 complete [ 2019.599896] LustreError: 77873:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2023.116900] LustreError: 76641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2023.124656] LustreError: 76641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2023.408296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.497851] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.621842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.661859] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2025.784813] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2028.235567] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2029.035826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2029.062260] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2029.107847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2033.639978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2033.647172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2033.657190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2036.987703] Lustre: server umount lustre-OST0000 complete [ 2044.386834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.388377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2044.392265] Lustre: Skipped 1 previous similar message [ 2044.393468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2044.405546] Lustre: Skipped 3 previous similar messages [ 2045.505332] Lustre: server umount lustre-MDT0000 complete [ 2047.370852] LustreError: 74783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362636 with bad export cookie 2522283779273568588 [ 2047.373393] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.376762] LustreError: 74783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2047.560360] Lustre: server umount lustre-MDT0001 complete [ 2051.997510] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2053.150420] Key type lgssc unregistered [ 2053.272561] LNet: 79663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.276553] LNetError: 79663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.289423] LNet: Removed LNI 192.168.201.156@tcp [ 2053.622428] Key type .llcrypt unregistered [ 2053.624079] Key type ._llcrypt unregistered [ 2058.364649] Key type ._llcrypt registered [ 2058.365929] Key type .llcrypt registered [ 2058.410270] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2058.534421] Key type .llcrypt unregistered [ 2058.535958] Key type ._llcrypt unregistered [ 2069.945056] Key type ._llcrypt registered [ 2069.946532] Key type .llcrypt registered [ 2069.996813] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2070.491259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2070.500132] alg: No test for adler32 (adler32-zlib) [ 2071.396458] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 2071.500191] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2073.088150] Key type lgssc registered [ 2073.535431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2076.904020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2076.913958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.094770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.644493] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2082.843508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.980125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2084.440521] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2087.432963] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2089.740102] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2093.054296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2093.222310] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2093.268615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.671547] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2102.787218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2104.113174] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2118.121925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2118.122040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.127995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2118.147104] Lustre: Skipped 1 previous similar message [ 2120.539805] Lustre: server umount lustre-OST0000 complete [ 2124.257741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2124.266112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.278646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.281120] Lustre: Skipped 1 previous similar message [ 2128.356154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2128.358636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2128.390720] Lustre: Skipped 1 previous similar message [ 2130.267390] Lustre: server umount lustre-MDT0000 complete [ 2133.473848] LustreError: 81540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2133.502485] LustreError: 81540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2133.710448] LustreError: 82274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362723 with bad export cookie 8803872319177530972 [ 2133.719809] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2133.730215] LustreError: 82274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2134.275943] Lustre: server umount lustre-MDT0001 complete [ 2143.893429] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2146.078454] Key type lgssc unregistered [ 2146.273622] LNet: 85071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.276640] LNetError: 85071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.291314] LNet: Removed LNI 192.168.201.156@tcp [ 2146.965305] Key type .llcrypt unregistered [ 2146.967370] Key type ._llcrypt unregistered [ 2162.205910] Key type ._llcrypt registered [ 2162.207735] Key type .llcrypt registered [ 2162.270313] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2162.957859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2163.015129] alg: No test for adler32 (adler32-zlib) [ 2163.980967] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 2164.151445] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2165.763083] Key type lgssc registered [ 2166.516781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2171.382913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2171.400215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.678156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.417903] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2180.716390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.943408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2183.157768] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2187.336913] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2190.371170] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2194.037162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2194.238379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.219023] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2198.385476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2201.732161] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2213.857992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2213.861750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.870868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2215.909158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2215.911760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2215.924161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.203747] Lustre: server umount lustre-OST0000 complete [ 2221.030149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.032236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2221.037851] Lustre: Skipped 1 previous similar message [ 2221.047548] Lustre: Skipped 2 previous similar messages [ 2225.627498] Lustre: server umount lustre-MDT0000 complete [ 2226.146803] LustreError: 86464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.154048] LustreError: 86464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2227.904619] LustreError: 86446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362817 with bad export cookie 15977651604418028142 [ 2227.908854] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2227.914701] LustreError: 86446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2228.101216] Lustre: server umount lustre-MDT0001 complete [ 2234.796328] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2236.657363] Key type lgssc unregistered [ 2236.879136] LNet: 89991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2236.889425] LNetError: 89991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.912551] LNet: Removed LNI 192.168.201.156@tcp [ 2237.521438] Key type .llcrypt unregistered [ 2237.523931] Key type ._llcrypt unregistered [ 2252.393083] Key type ._llcrypt registered [ 2252.395386] Key type .llcrypt registered [ 2252.456358] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2253.000403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2253.110526] alg: No test for adler32 (adler32-zlib) [ 2254.063658] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 2254.244372] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2255.864163] Key type lgssc registered [ 2256.430949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2261.234839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2261.251759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.560523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.102231] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2271.059273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.402376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2274.543960] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2279.134598] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2282.428669] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2286.637132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.903981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.931755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2290.645297] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2295.219123] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2303.466270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2303.472392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.487515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.530693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2306.532046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.532927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2308.740593] Lustre: server umount lustre-OST0000 complete [ 2311.655106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.664837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.668135] Lustre: Skipped 1 previous similar message [ 2311.674391] Lustre: Skipped 2 previous similar messages [ 2316.771050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.777040] Lustre: Skipped 1 previous similar message [ 2317.102112] Lustre: server umount lustre-MDT0000 complete [ 2319.560294] LustreError: 94361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362909 with bad export cookie 9135483567252815197 [ 2319.561561] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.574403] LustreError: 94361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2319.853371] Lustre: server umount lustre-MDT0001 complete [ 2326.870997] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2328.752680] Key type lgssc unregistered [ 2328.906803] LNet: 94910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.910848] LNetError: 94910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.920794] LNet: Removed LNI 192.168.201.156@tcp [ 2329.477729] Key type .llcrypt unregistered [ 2329.485386] Key type ._llcrypt unregistered [ 2343.975989] Key type ._llcrypt registered [ 2343.978180] Key type .llcrypt registered [ 2344.056247] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2344.602772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.676089] alg: No test for adler32 (adler32-zlib) [ 2345.617032] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 2345.813079] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2347.416284] Key type lgssc registered [ 2348.194446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2353.347688] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2353.361931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2368.865790] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2373.984374] LustreError: 96274:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 2400.608217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.454665] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2409.692674] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2413.473892] Lustre: server umount lustre-OST0000 complete [ 2419.254511] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 11:30:08 (1788363008) [ 2424.164492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2424.404778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2426.941632] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2429.416742] LustreError: 97912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.426313] LustreError: 97912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2432.087663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.318652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2434.592241] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2438.926423] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2442.482669] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2447.350207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2447.635616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.705229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2452.861473] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2458.621542] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2467.298227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.314185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2467.318299] Lustre: Skipped 1 previous similar message [ 2472.420498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2472.426775] Lustre: Skipped 1 previous similar message [ 2473.414639] Lustre: server umount lustre-OST0000 complete [ 2477.539940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.554901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2477.568382] Lustre: Skipped 2 previous similar messages [ 2482.665554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2482.671699] Lustre: Skipped 2 previous similar messages [ 2487.778840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.786503] Lustre: Skipped 1 previous similar message [ 2490.337091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2490.522832] Lustre: server umount lustre-MDT0000 complete [ 2492.899444] LustreError: 97912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.913524] LustreError: 97912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2493.153380] LustreError: 97892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363082 with bad export cookie 17395081938092784474 [ 2493.155718] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.163647] LustreError: 97892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2493.405989] Lustre: server umount lustre-MDT0001 complete [ 2501.487872] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2503.332954] Key type lgssc unregistered [ 2503.573976] LNet: 101603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2503.578511] LNetError: 101603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2503.588728] LNet: Removed LNI 192.168.201.156@tcp [ 2504.280427] Key type .llcrypt unregistered [ 2504.282137] Key type ._llcrypt unregistered [ 2533.772722] Key type ._llcrypt registered [ 2533.777369] Key type .llcrypt registered [ 2533.857549] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2534.802796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2534.837635] alg: No test for adler32 (adler32-zlib) [ 2535.884713] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 2536.086992] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2537.736215] Key type lgssc registered [ 2538.566361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2543.937579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2543.949179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.322049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.646925] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2554.860667] LustreError: 103768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.883618] LustreError: 103768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2555.372653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.613193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2559.708865] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2566.001454] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2570.325706] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2574.852926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.056700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.590459] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2584.523770] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2590.450198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2600.936083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2600.939868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.946521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2600.955784] Lustre: Skipped 1 previous similar message [ 2600.964167] Lustre: Skipped 1 previous similar message [ 2605.487359] Lustre: server umount lustre-OST0000 complete [ 2611.169676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.174491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2611.180266] Lustre: Skipped 1 previous similar message [ 2611.190322] Lustre: Skipped 1 previous similar message [ 2614.699077] Lustre: server umount lustre-MDT0000 complete [ 2616.289730] LustreError: 103764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2616.300488] LustreError: 103764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2617.066852] LustreError: 105616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363206 with bad export cookie 10360613112910422267 [ 2617.068279] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2617.074366] LustreError: 105616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2617.294595] Lustre: server umount lustre-MDT0001 complete [ 2624.305944] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2626.243505] Key type lgssc unregistered [ 2626.529537] LNet: 107327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2626.535862] LNetError: 107327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2626.551553] LNet: Removed LNI 192.168.201.156@tcp [ 2627.257821] Key type .llcrypt unregistered [ 2627.259030] Key type ._llcrypt unregistered [ 2639.370421] Key type ._llcrypt registered [ 2639.375762] Key type .llcrypt registered [ 2639.433964] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2639.892733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2639.924203] alg: No test for adler32 (adler32-zlib) [ 2640.868872] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 2641.039698] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2642.680353] Key type lgssc registered [ 2643.287653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2648.428327] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2648.442597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2663.968409] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2669.088322] LustreError: 108548:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 2695.778045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2700.070238] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2704.882283] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2709.731420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.891650] LustreError: MGC192.168.201.156@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2709.898399] LustreError: 109570:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2709.903516] LustreError: 109570:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2709.906574] LustreError: 109570:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2709.947837] Lustre: server umount lustre-MDT0000 complete [ 2709.949737] LustreError: 109570:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2725.419853] Lustre: server umount lustre-OST0000 complete [ 2731.263621] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 2737.308777] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2761.283905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.456553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2761.474136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2761.517738] Lustre: lustre-MDT0000: new disk, initializing [ 2761.576515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.593623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2764.089943] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2771.394926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.503326] Lustre: 112359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2771.553983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2771.558133] Lustre: Skipped 1 previous similar message [ 2771.619528] Lustre: lustre-MDT0001: new disk, initializing [ 2771.692854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2771.716885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2771.729476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2774.398733] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2780.721323] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2783.947453] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2787.941730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.106827] Lustre: lustre-OST0000: new disk, initializing [ 2788.109485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2788.113693] Lustre: 114327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2788.187399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.677479] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2793.499032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2793.510116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2793.575827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2797.787799] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2800.715348] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2800.850981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2804.019860] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2804.182182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2808.811554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.825742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2808.826356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2808.846043] Lustre: Skipped 1 previous similar message [ 2811.732888] Lustre: server umount lustre-OST0000 complete [ 2818.020657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2818.024674] LustreError: Skipped 1 previous similar message [ 2818.026413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.040951] Lustre: Skipped 1 previous similar message [ 2818.044423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2819.047551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.058241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2819.062379] Lustre: Skipped 1 previous similar message [ 2820.450793] Lustre: server umount lustre-MDT0000 complete [ 2823.001824] LustreError: 112349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363412 with bad export cookie 9023638899939326201 [ 2823.006369] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2823.008541] LustreError: 112349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2823.297814] Lustre: server umount lustre-MDT0001 complete [ 2828.379241] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 11:36:57 (1788363417) [ 2830.134067] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2838.722396] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 2844.016202] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2864.944275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.166573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2865.193861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2865.241241] Lustre: lustre-MDT0000: new disk, initializing [ 2865.312974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.324706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2868.182492] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2874.945418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.031714] Lustre: 119221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2875.070717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2875.074218] Lustre: Skipped 1 previous similar message [ 2875.124263] Lustre: lustre-MDT0001: new disk, initializing [ 2875.190400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2875.217935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2875.225746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2877.735359] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2883.640313] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2887.202400] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2891.316762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.469790] Lustre: lustre-OST0000: new disk, initializing [ 2891.472104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2891.476483] Lustre: 121188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2891.530453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.499399] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2899.997767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2900.002435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2900.091946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2900.230572] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2902.937780] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2903.067496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2905.899369] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2906.049660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2910.177904] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2910.180261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2910.182942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2910.194203] Lustre: Skipped 1 previous similar message [ 2910.199329] Lustre: Skipped 1 previous similar message [ 2913.371567] Lustre: server umount lustre-OST0000 complete [ 2916.324578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2916.334332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.345653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.703385] Lustre: server umount lustre-MDT0000 complete [ 2923.512385] LustreError: 119213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363513 with bad export cookie 9023638899939327083 [ 2923.514541] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2923.518245] LustreError: 119213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2923.666795] Lustre: server umount lustre-MDT0001 complete [ 2928.377148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.677724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.120218] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2933.732702] LustreError: 123403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.743073] LustreError: 123403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2936.517822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.582318] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2943.845258] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2947.024705] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2950.854894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2954.288404] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2958.254754] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2966.293461] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2969.570973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2969.578830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.586078] Lustre: Skipped 1 previous similar message [ 2969.589321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2969.592845] Lustre: Skipped 2 previous similar messages [ 2970.133264] Lustre: server umount lustre-OST0000 complete [ 2978.507689] Lustre: server umount lustre-MDT0000 complete [ 2980.372614] LustreError: 123384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363569 with bad export cookie 9023638899939327972 [ 2980.376526] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.378466] LustreError: 123384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2989.627291] Lustre: server umount lustre-MDT0001 complete [ 2995.712378] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2997.404719] Key type lgssc unregistered [ 2997.582686] LNet: 127027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2997.586497] LNetError: 127027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2997.603434] LNet: Removed LNI 192.168.201.156@tcp [ 2998.131668] Key type .llcrypt unregistered [ 2998.134649] Key type ._llcrypt unregistered [ 3008.227852] Key type ._llcrypt registered [ 3008.232131] Key type .llcrypt registered [ 3008.295593] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 3016.262095] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3016.751768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3016.765733] alg: No test for adler32 (adler32-zlib) [ 3017.643537] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 3017.750913] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3019.336178] Key type lgssc registered [ 3019.920801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3036.766435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3036.773970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.881897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3037.897843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3037.954492] Lustre: lustre-MDT0000: new disk, initializing [ 3038.014066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.022472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3040.124104] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3045.550076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.593246] Lustre: 130155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3045.633738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3045.636490] Lustre: Skipped 1 previous similar message [ 3045.680792] Lustre: lustre-MDT0001: new disk, initializing [ 3045.728536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3045.748473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3045.753848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3047.774223] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3052.366814] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3054.856174] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3057.690274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3057.836154] Lustre: lustre-OST0000: new disk, initializing [ 3057.839285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3057.842885] Lustre: 132124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3057.885931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.332861] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3064.341673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3064.347974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3064.393147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3064.532226] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3066.130457] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3066.208588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3068.019441] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3068.107151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3069.409127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3069.410571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.411567] LustreError: Skipped 1 previous similar message [ 3069.416529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3069.419597] Lustre: Skipped 1 previous similar message [ 3070.936251] Lustre: server umount lustre-OST0000 complete [ 3074.531807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3074.537295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3074.539831] Lustre: Skipped 2 previous similar messages [ 3078.755243] Lustre: server umount lustre-MDT0000 complete [ 3080.465649] LustreError: 130145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363669 with bad export cookie 16687506033237520670 [ 3080.468417] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3080.470596] LustreError: 130145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3080.644661] Lustre: server umount lustre-MDT0001 complete [ 3084.760143] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 11:41:13 (1788363673) [ 3087.488800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.719895] LustreError: 134532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3087.727857] LustreError: 134532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3087.750363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.821392] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3092.963741] LustreError: 134533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3093.892137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.077213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3096.091405] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3099.287678] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3101.651284] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3103.201955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.202640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.207646] Lustre: Skipped 1 previous similar message [ 3103.210023] Lustre: Skipped 1 previous similar message [ 3108.629057] Lustre: server umount lustre-MDT0000 complete [ 3109.344967] LustreError: 134551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.350931] LustreError: 134551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3110.312771] LustreError: 134513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363699 with bad export cookie 16687506033237521538 [ 3110.313080] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3110.317724] LustreError: 134513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3110.459739] Lustre: server umount lustre-MDT0001 complete [ 3113.584299] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 11:41:42 (1788363702) [ 3116.262268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3131.744632] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3136.864360] LustreError: 136944:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 3162.464358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3164.593926] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3167.304543] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3168.077721] Lustre: server umount lustre-OST0000 complete [ 3170.964621] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 11:42:40 (1788363760) [ 3173.474289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.667176] LustreError: 138386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.211651] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3178.454701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.637659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3178.645114] Lustre: Skipped 1 previous similar message [ 3180.241504] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3183.120058] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3185.156271] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3187.752389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3190.035329] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3192.686366] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3209.701379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3209.701960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3209.702370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3209.702381] Lustre: Skipped 3 previous similar messages [ 3209.707226] Lustre: Skipped 1 previous similar message [ 3211.353594] Lustre: server umount lustre-MDT0000 complete [ 3212.725338] LustreError: 138368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363802 with bad export cookie 16687506033237522189 [ 3212.728641] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3212.729905] LustreError: 138368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3212.849689] Lustre: server umount lustre-MDT0001 complete [ 3224.424904] Lustre: server umount lustre-OST0000 complete [ 3227.550595] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 11:43:36 (1788363816) [ 3230.098376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3230.287151] LustreError: 142058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.318618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3230.321585] Lustre: Skipped 1 previous similar message [ 3231.956688] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3235.364163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.577295] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3241.215215] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3243.908767] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3247.175350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3250.137879] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3253.554502] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3254.509533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3255.417194] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3255.496062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3257.027645] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3261.241765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 3264.992940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3264.996110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.001316] Lustre: Skipped 1 previous similar message [ 3265.003777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3265.006039] Lustre: Skipped 2 previous similar messages [ 3268.319604] Lustre: server umount lustre-OST0000 complete [ 3276.125674] Lustre: server umount lustre-MDT0000 complete [ 3276.257746] LustreError: 142054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.264650] LustreError: 142054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3277.727309] LustreError: 142040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363867 with bad export cookie 16687506033237523246 [ 3277.730035] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3277.731981] LustreError: 142040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3277.873173] Lustre: server umount lustre-MDT0001 complete [ 3281.287086] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 11:44:30 (1788363870) [ 3284.226263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3299.680743] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3304.800368] LustreError: 146238:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 3330.400310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.405144] Lustre: Skipped 2 previous similar messages [ 3332.847324] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3335.870227] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3338.907633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.136941] LustreError: 146263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.210654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3340.857735] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3344.897795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.761801] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3350.191441] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3352.729903] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3354.686327] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3354.794863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3356.450142] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3356.556970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3360.738800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3360.742125] LustreError: Skipped 1 previous similar message [ 3360.743894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3360.748939] Lustre: Skipped 3 previous similar messages [ 3360.752272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3360.755251] Lustre: Skipped 4 previous similar messages [ 3363.562158] Lustre: server umount lustre-MDT0000 complete [ 3365.130064] LustreError: 146246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363954 with bad export cookie 16687506033237524254 [ 3365.132086] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3365.136253] LustreError: 146246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3365.275101] Lustre: server umount lustre-MDT0001 complete [ 3377.063550] Lustre: server umount lustre-OST0000 complete [ 3380.492382] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 11:46:09 (1788363969) [ 3383.574157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3399.009225] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3404.128470] LustreError: 150425:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 3432.243308] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3435.613635] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3438.590775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.797271] LustreError: 150450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.810164] LustreError: 150450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3438.872648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3440.738825] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3444.797368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3446.670407] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3449.934189] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3452.479711] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3455.301809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3455.377496] Lustre: lustre-OST0001: new disk, initializing [ 3455.379619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3455.383603] Lustre: 153301:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3457.786162] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3462.178159] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3464.165744] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3464.705988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3464.709984] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3464.756738] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3465.284735] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3466.977409] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3467.065816] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3469.793513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3469.798181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.801500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.803055] Lustre: Skipped 3 previous similar messages [ 3469.805509] Lustre: Skipped 3 previous similar messages [ 3482.080168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3482.122446] Lustre: server umount lustre-OST0000 complete [ 3485.157425] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3485.160758] LustreError: Skipped 1 previous similar message [ 3489.824716] Lustre: server umount lustre-OST0001 complete [ 3497.578064] Lustre: server umount lustre-MDT0000 complete [ 3499.394667] LustreError: 150433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364088 with bad export cookie 16687506033237525206 [ 3499.395705] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3499.399924] LustreError: 150433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3499.549414] Lustre: server umount lustre-MDT0001 complete [ 3509.913173] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 11:48:19 (1788364099) [ 3510.653813] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3511.370755] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3512.102593] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 11:48:21 (1788364101) [ 3515.040217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3515.205380] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3515.218890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3515.304117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.307255] Lustre: Skipped 6 previous similar messages [ 3517.262989] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3521.008146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3521.060152] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3521.073448] Lustre: 156867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3522.860203] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3525.721429] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3528.193524] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3530.914901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3530.984780] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3533.387964] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3534.303082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3536.321464] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3538.066547] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3538.190444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3539.825667] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3539.913923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3544.548411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3544.548914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3544.549653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3544.549658] Lustre: Skipped 9 previous similar messages [ 3544.551249] LustreError: Skipped 1 previous similar message [ 3544.555304] Lustre: Skipped 5 previous similar messages [ 3546.843442] Lustre: server umount lustre-OST0000 complete [ 3561.270704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3563.834261] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3564.522582] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3564.527540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3564.527826] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3564.537306] Lustre: Skipped 1 previous similar message [ 3567.018869] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3568.878335] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3568.964643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3570.653402] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3570.763794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3573.101810] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e3f0674f000.ost_server_uuid 50 [ 3573.744733] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e3f0674f000.ost_server_uuid in FULL state after 0 sec [ 3573.826039] LustreError: 156874:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.56@tcp) returned error from blocking AST (req@00000000dac2ce37 x1875235157772032 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000082a12889/0xe795ee52b24f8c22 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.56@tcp remote: 0x3f487fe7b6a1a499 expref: 5 pid: 158042 timeout: 3673 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3573.837036] LustreError: lustre-MDT0000: A client on nid 192.168.201.56@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3573.839862] LustreError: 156863:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.56@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000082a12889/0xe795ee52b24f8c22 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.56@tcp remote: 0x3f487fe7b6a1a499 expref: 6 pid: 158042 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3579.875853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3579.883207] LustreError: Skipped 2 previous similar messages [ 3581.076137] Lustre: server umount lustre-OST0000 complete [ 3588.707888] Lustre: server umount lustre-MDT0000 complete [ 3590.116241] LustreError: 158044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.122976] LustreError: 158044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3590.333150] LustreError: 156859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364179 with bad export cookie 16687506033237526389 [ 3590.333993] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3590.339852] LustreError: 156859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3596.537630] Lustre: server umount lustre-MDT0001 complete [ 3600.821883] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3601.975201] Key type lgssc unregistered [ 3602.112347] LNet: 162970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3602.114891] LNetError: 162970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3602.123453] LNet: Removed LNI 192.168.201.156@tcp [ 3602.498256] Key type .llcrypt unregistered [ 3602.499672] Key type ._llcrypt unregistered [ 3612.568364] Key type ._llcrypt registered [ 3612.569944] Key type .llcrypt registered [ 3612.627402] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3613.152464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3613.174266] alg: No test for adler32 (adler32-zlib) [ 3614.055393] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 3614.153496] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3615.744151] Key type lgssc registered [ 3616.177132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3619.437429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3619.445641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.610109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.148785] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3625.617393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3625.766978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3627.338325] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3630.100961] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3632.455509] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3635.216874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3635.353639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.503340] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3640.530534] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3645.611285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3648.192328] Lustre: Failing over lustre-MDT0000 [ 3648.324212] Lustre: server umount lustre-MDT0000 complete [ 3651.041627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.043213] LustreError: 164354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.049118] Lustre: Skipped 2 previous similar messages [ 3651.053422] LustreError: 164354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3656.162239] LustreError: 164356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3656.170504] LustreError: 164356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3659.279580] LustreError: 164385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.56@tcp arrived at 1788364248 with bad export cookie 15515278094639239223 [ 3659.284633] LustreError: 164385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3661.281433] LustreError: 166849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.289072] LustreError: 166849:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3662.106754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3662.152105] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3662.258172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.295463] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3663.907247] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3664.834142] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3664.838030] Lustre: lustre-MDT0000: Denying connection for new client 4e55d8cb-da21-426a-9949-ef26027642e2 (at 192.168.201.56@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3667.439152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3672.549461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3672.549590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.550455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3672.561259] Lustre: Skipped 1 previous similar message [ 3677.666916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.669080] Lustre: Skipped 2 previous similar messages [ 3677.720062] Lustre: server umount lustre-OST0000 complete [ 3679.062506] LustreError: 168087:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3679.067761] Lustre: 167437:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3679.071514] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3679.078363] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3679.082730] LustreError: 167437:0:(client.c:1384:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d5f1c268 x1875235782560256/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3679.089828] LustreError: 167437:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3679.093428] LustreError: 167437:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3679.096939] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 3679.213399] Lustre: server umount lustre-MDT0000 complete [ 3680.528243] LustreError: 164341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364270 with bad export cookie 15515278094639239552 [ 3680.534677] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3680.535260] LustreError: 166849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.721883] Lustre: server umount lustre-MDT0001 complete [ 3690.746180] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3691.882952] Key type lgssc unregistered [ 3692.024319] LNet: 168839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3692.027135] LNetError: 168839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3692.033345] LNet: Removed LNI 192.168.201.156@tcp [ 3692.387885] Key type .llcrypt unregistered [ 3692.389425] Key type ._llcrypt unregistered [ 3702.423299] Key type ._llcrypt registered [ 3702.424588] Key type .llcrypt registered [ 3702.474369] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3702.939471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3702.950994] alg: No test for adler32 (adler32-zlib) [ 3703.845515] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 3703.961931] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3705.560205] Key type lgssc registered [ 3705.978154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3709.030563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3709.039324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.226274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3712.125282] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3715.775588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3715.922382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3717.584146] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3720.634760] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3722.940595] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3725.852497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3726.022523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3728.314330] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3730.866629] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3735.271196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3737.569576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3737.574078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.580118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3740.643029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3740.644440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3740.649809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.575781] Lustre: server umount lustre-OST0000 complete [ 3745.761917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.767334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3745.769764] Lustre: Skipped 2 previous similar messages [ 3750.881615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3750.885113] Lustre: Skipped 1 previous similar message [ 3751.192388] Lustre: server umount lustre-MDT0000 complete [ 3752.553657] LustreError: 170210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364342 with bad export cookie 4444999036361233756 [ 3752.556244] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3752.557479] LustreError: 170210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3752.689797] Lustre: server umount lustre-MDT0001 complete [ 3756.607386] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3757.821346] Key type lgssc unregistered [ 3757.993504] LNet: 173758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3757.998221] LNetError: 173758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3758.007551] LNet: Removed LNI 192.168.201.156@tcp [ 3758.377439] Key type .llcrypt unregistered [ 3758.378725] Key type ._llcrypt unregistered [ 3772.384114] Key type ._llcrypt registered [ 3772.385298] Key type .llcrypt registered [ 3772.437294] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3772.929810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3772.946518] alg: No test for adler32 (adler32-zlib) [ 3773.830380] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 3773.931485] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3775.520192] Key type lgssc registered [ 3775.951623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3779.010652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3779.016850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.195903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.979163] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3785.674584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3785.904187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3787.515832] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3790.480867] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3792.573044] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3795.564505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3795.723820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.273963] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3801.867912] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3804.971112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3810.275140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.275595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3810.276299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3810.280826] Lustre: Skipped 1 previous similar message [ 3811.876792] Lustre: server umount lustre-OST0000 complete [ 3815.395753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3815.400240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3815.416235] Lustre: Skipped 1 previous similar message [ 3815.421291] Lustre: Skipped 2 previous similar messages [ 3815.783891] Lustre: server umount lustre-MDT0000 complete [ 3817.586314] LustreError: 177929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364407 with bad export cookie 4967963449857279230 [ 3817.589588] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3817.590182] LustreError: 177929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3817.753126] Lustre: server umount lustre-MDT0001 complete [ 3823.614698] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3825.365228] Key type lgssc unregistered [ 3825.580240] LNet: 178966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3825.587234] LNetError: 178966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3825.610433] LNet: Removed LNI 192.168.201.156@tcp [ 3826.105545] Key type .llcrypt unregistered [ 3826.107148] Key type ._llcrypt unregistered [ 3840.167707] Key type ._llcrypt registered [ 3840.169486] Key type .llcrypt registered [ 3840.268817] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3840.937396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3840.980333] alg: No test for adler32 (adler32-zlib) [ 3842.025948] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 3842.202795] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3843.816211] Key type lgssc registered [ 3844.585625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3851.720917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3851.741884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.012239] Lustre: *** cfs_fail_loc=135, val=0*** [ 3853.074117] LustreError: 180467:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3853.078278] LustreError: 180467:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.156@tcp: cfg command failed: rc = -2 [ 3853.088174] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3853.088174] [ 3853.093993] LustreError: MGC192.168.201.156@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3853.103033] LustreError: 180428:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3853.109770] LustreError: 180428:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3853.120490] LustreError: 180428:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3853.124937] LustreError: 180428:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3853.168592] Lustre: server umount lustre-MDT0000 complete [ 3853.171914] LustreError: 180428:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3857.916147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.168630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.456910] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3865.022716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3865.243538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3867.184436] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3870.624618] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3873.462779] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3875.490230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3875.498344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3875.501446] Lustre: Skipped 1 previous similar message [ 3880.633337] Lustre: server umount lustre-MDT0000 complete [ 3880.930990] LustreError: 180839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.938742] LustreError: 180839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3882.576524] LustreError: 180818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364472 with bad export cookie 15895874498779158898 [ 3882.579596] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3882.583781] LustreError: 180818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3882.774655] Lustre: server umount lustre-MDT0001 complete [ 3887.774636] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3889.113742] Key type lgssc unregistered [ 3889.308168] LNet: 183157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3889.314944] LNetError: 183157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3889.325809] LNet: Removed LNI 192.168.201.156@tcp [ 3889.774702] Key type .llcrypt unregistered [ 3889.776526] Key type ._llcrypt unregistered [ 3900.960585] Key type ._llcrypt registered [ 3900.961709] Key type .llcrypt registered [ 3901.008635] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3901.160341] Key type .llcrypt unregistered [ 3901.161653] Key type ._llcrypt unregistered [ 3910.604712] Key type ._llcrypt registered [ 3910.606477] Key type .llcrypt registered [ 3910.687959] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3911.329557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3911.348431] alg: No test for adler32 (adler32-zlib) [ 3912.282712] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 3912.450857] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3914.072176] Key type lgssc registered [ 3914.611705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3918.860775] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3918.870220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3934.369696] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3939.488320] LustreError: 185266:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 3966.117727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3968.728107] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3972.049483] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3975.602197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.940406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3975.985171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3978.227476] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3981.090925] LustreError: 185291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3981.103562] LustreError: 185291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3982.206637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3982.417944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3984.276517] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3987.238082] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3989.564208] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3990.719059] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3996.641068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3996.644087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.651842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3997.665637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3997.667486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3997.668459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4002.199240] Lustre: server umount lustre-OST0000 complete [ 4007.906033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4007.906553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.907544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4007.907548] Lustre: Skipped 1 previous similar message [ 4007.920134] Lustre: Skipped 1 previous similar message [ 4013.025348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4013.028049] Lustre: Skipped 2 previous similar messages [ 4018.144131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4018.145387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.150690] Lustre: Skipped 1 previous similar message [ 4018.203623] Lustre: server umount lustre-MDT0000 complete [ 4019.628506] LustreError: 187926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364609 with bad export cookie 16594015444860444205 [ 4019.630833] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4019.633285] LustreError: 187926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4019.764089] Lustre: server umount lustre-MDT0001 complete [ 4024.024776] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4025.331645] Key type lgssc unregistered [ 4025.485660] LNet: 189181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4025.489257] LNetError: 189181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4025.499567] LNet: Removed LNI 192.168.201.156@tcp [ 4025.847158] Key type .llcrypt unregistered [ 4025.848853] Key type ._llcrypt unregistered [ 4037.179500] Key type ._llcrypt registered [ 4037.180872] Key type .llcrypt registered [ 4037.232654] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4037.782869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4037.792823] alg: No test for adler32 (adler32-zlib) [ 4038.676906] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4038.801793] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4040.400235] Key type lgssc registered [ 4040.974034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4044.848742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4044.857978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.108168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4047.860355] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4051.602215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.810931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4053.539140] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4056.588833] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4059.122632] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4062.222560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4062.387454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4064.771736] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4067.886754] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4070.571047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4076.679252] Lustre: Failing over lustre-MDT0000 [ 4076.805672] Lustre: server umount lustre-MDT0000 complete [ 4077.540555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4077.544298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.551747] LustreError: 190572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.572712] LustreError: 190572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4078.565149] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.569931] LustreError: 191810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.576704] Lustre: Skipped 1 previous similar message [ 4078.600915] LustreError: 191810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4080.349851] LustreError: 191263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4083.683870] LustreError: 190571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4083.690053] LustreError: 190571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4088.802137] LustreError: 190572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.808320] LustreError: 190572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4090.994917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.056400] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4091.159705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.201956] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4092.738574] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4094.543368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4095.689544] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4096.486187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4096.493253] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4096.513622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4096.685187] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4106.727025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4106.729296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4106.738842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4106.742562] Lustre: Skipped 1 previous similar message [ 4111.514917] Lustre: server umount lustre-OST0000 complete [ 4116.963098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4116.963746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.966525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4116.970934] Lustre: Skipped 2 previous similar messages [ 4116.980410] Lustre: Skipped 1 previous similar message [ 4119.225443] Lustre: server umount lustre-MDT0000 complete [ 4120.912256] LustreError: 192414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364710 with bad export cookie 1796940019314017997 [ 4120.917962] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4120.919591] LustreError: 192414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4121.086717] Lustre: server umount lustre-MDT0001 complete [ 4126.882868] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4128.305309] Key type lgssc unregistered [ 4128.454663] LNet: 195353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4128.458498] LNetError: 195353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4128.475124] LNet: Removed LNI 192.168.201.156@tcp [ 4128.876898] Key type .llcrypt unregistered [ 4128.878338] Key type ._llcrypt unregistered [ 4139.919054] Key type ._llcrypt registered [ 4139.920407] Key type .llcrypt registered [ 4139.961639] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4140.393802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4140.404316] alg: No test for adler32 (adler32-zlib) [ 4141.280480] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4141.386478] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4142.992258] Key type lgssc registered [ 4143.498912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4146.707653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4146.718584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4147.988223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4149.798743] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4153.067540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.264494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4154.940329] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4158.091991] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4160.552755] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4163.427041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4163.493033] LustreError: 198560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4163.497788] LustreError: 198560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4163.584242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.924294] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4168.891256] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4172.776833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4174.978531] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4179.539199] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4191.860625] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4198.370394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4198.371775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.374914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4198.378536] Lustre: Skipped 1 previous similar message [ 4198.382769] Lustre: Skipped 1 previous similar message [ 4203.107987] Lustre: server umount lustre-OST0000 complete [ 4208.609488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.610222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.613794] Lustre: Skipped 1 previous similar message [ 4210.659900] Lustre: server umount lustre-MDT0000 complete [ 4212.019241] LustreError: 199279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364801 with bad export cookie 13395559217316903771 [ 4212.022704] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4212.024413] LustreError: 199279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4212.142815] Lustre: server umount lustre-MDT0001 complete [ 4215.954033] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4217.043755] Key type lgssc unregistered [ 4217.185438] LNet: 200606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4217.189106] LNetError: 200606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4217.198358] LNet: Removed LNI 192.168.201.156@tcp [ 4217.531320] Key type .llcrypt unregistered [ 4217.533080] Key type ._llcrypt unregistered [ 4225.135789] Key type ._llcrypt registered [ 4225.137163] Key type .llcrypt registered [ 4225.178052] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4225.547627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4225.578509] alg: No test for adler32 (adler32-zlib) [ 4226.447653] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4226.548347] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4228.136268] Key type lgssc registered [ 4228.540542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4231.620661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4231.629646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4232.867878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.546395] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4237.794630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.973181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4239.594923] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4242.489320] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4244.784523] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4247.331580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4247.494912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4250.011828] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4252.952881] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4257.769966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4260.275684] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4270.152598] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4276.239148] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4280.686610] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4289.506038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4289.511777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.523479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4291.556518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4291.556600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4291.565072] Lustre: Skipped 1 previous similar message [ 4295.659792] Lustre: server umount lustre-OST0000 complete [ 4299.744920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4299.748127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.755296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4301.793021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4301.794254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.324821] Lustre: server umount lustre-MDT0000 complete [ 4304.755409] LustreError: 201838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364894 with bad export cookie 16839562576177704581 [ 4304.758649] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4304.758908] LustreError: 201838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4304.871286] Lustre: server umount lustre-MDT0001 complete [ 4308.683408] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4309.774301] Key type lgssc unregistered [ 4309.900367] LNet: 207076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4309.903236] LNetError: 207076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4309.912322] LNet: Removed LNI 192.168.201.156@tcp [ 4310.249072] Key type .llcrypt unregistered [ 4310.250618] Key type ._llcrypt unregistered [ 4319.866292] Key type ._llcrypt registered [ 4319.867615] Key type .llcrypt registered [ 4319.911990] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4320.242811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4320.255887] alg: No test for adler32 (adler32-zlib) [ 4321.123667] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4321.223481] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4322.808174] Key type lgssc registered [ 4323.306259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4326.337604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4326.344517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4327.547526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4329.128190] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4332.379246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.557311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4334.089479] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4336.735339] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4338.626382] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4340.928259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4341.056706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4343.295216] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4346.021330] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4354.411718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4363.744955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4363.747852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.752408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4364.770306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4364.772231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4364.778438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4366.549339] Lustre: server umount lustre-OST0000 complete [ 4368.353173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4368.356611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.363331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4368.365623] Lustre: Skipped 1 previous similar message [ 4374.105497] Lustre: server umount lustre-MDT0000 complete [ 4375.009056] LustreError: 208466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.013435] LustreError: 208466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4375.398084] LustreError: 208447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364964 with bad export cookie 14552468879505012426 [ 4375.401790] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4375.403167] LustreError: 208447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4375.515678] Lustre: server umount lustre-MDT0001 complete [ 4379.462942] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4380.551022] Key type lgssc unregistered [ 4380.683404] LNet: 212167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4380.685655] LNetError: 212167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4380.692351] LNet: Removed LNI 192.168.201.156@tcp [ 4381.003121] Key type .llcrypt unregistered [ 4381.004680] Key type ._llcrypt unregistered [ 4388.714332] Key type ._llcrypt registered [ 4388.715758] Key type .llcrypt registered [ 4388.754718] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4389.095388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4389.119032] alg: No test for adler32 (adler32-zlib) [ 4389.972652] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4390.069598] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4391.656160] Key type lgssc registered [ 4392.021302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4394.677770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4394.685182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.840533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.382272] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4400.627039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4400.771296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4402.227983] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4405.032366] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4407.260359] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4409.982571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4410.172799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4412.687636] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4415.495334] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4421.417346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4426.722965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.724056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4426.728446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4426.732879] Lustre: Skipped 1 previous similar message [ 4427.157838] Lustre: server umount lustre-OST0000 complete [ 4430.612776] Lustre: server umount lustre-MDT0000 complete [ 4431.841359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4431.841618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4431.850473] Lustre: Skipped 1 previous similar message [ 4431.852569] LustreError: 214142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4431.857893] LustreError: 214142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4431.892323] LustreError: 213397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365021 with bad export cookie 15080427701274361973 [ 4431.894916] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4431.898379] LustreError: 213397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4432.010518] Lustre: server umount lustre-MDT0001 complete [ 4435.792292] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4436.872827] Key type lgssc unregistered [ 4437.001319] LNet: 217091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4437.003915] LNetError: 217091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4437.015315] LNet: Removed LNI 192.168.201.156@tcp [ 4437.326955] Key type .llcrypt unregistered [ 4437.328243] Key type ._llcrypt unregistered [ 4444.842676] Key type ._llcrypt registered [ 4444.843815] Key type .llcrypt registered [ 4444.882511] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4445.226721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4445.254590] alg: No test for adler32 (adler32-zlib) [ 4446.106184] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4446.188762] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4447.770201] Key type lgssc registered [ 4448.169830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4450.854204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4450.862778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.009808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.570272] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4456.548251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.675191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4458.069250] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4460.485986] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4462.289051] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4464.597237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4464.732132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.586728] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4469.005686] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4477.031182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4482.340995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4482.377063] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4482.410975] LustreError: 220151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.417217] LustreError: 220151:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4482.467961] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4484.476993] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4486.931761] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4488.448644] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 4499.049803] Lustre: Permanently deactivating lustre-OST0001 [ 4499.055210] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4507.654192] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4512.264901] Lustre: Permanently reactivating lustre-OST0001 [ 4512.270979] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4512.273282] Lustre: Skipped 2 previous similar messages [ 4514.883402] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.889239] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4514.892683] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4514.897933] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4514.910531] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4519.904734] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4519.906939] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.910278] Lustre: Skipped 1 previous similar message [ 4519.912177] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4520.416933] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4520.420890] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4522.978913] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4522.981548] Lustre: Skipped 1 previous similar message [ 4525.571931] Lustre: server umount lustre-OST0001 complete [ 4528.099298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.100292] LustreError: 220152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.100766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4528.105051] Lustre: Skipped 2 previous similar messages [ 4528.110121] LustreError: 220152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4533.217980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4533.219776] LustreError: 220153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.220186] Lustre: Skipped 2 previous similar messages [ 4538.342851] LustreError: 220151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4538.350819] LustreError: 220151:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4541.408161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4541.459018] Lustre: server umount lustre-OST0000 complete [ 4543.457867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.458577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4543.462412] Lustre: Skipped 1 previous similar message [ 4543.466418] Lustre: Skipped 3 previous similar messages [ 4548.840672] Lustre: server umount lustre-MDT0000 complete [ 4550.104254] LustreError: 221190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365139 with bad export cookie 17199595305884146266 [ 4550.107629] LustreError: 221190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4550.107947] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4550.240119] Lustre: server umount lustre-MDT0001 complete [ 4553.878541] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4554.970263] Key type lgssc unregistered [ 4555.106398] LNet: 223484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4555.108938] LNetError: 223484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4555.116279] LNet: Removed LNI 192.168.201.156@tcp [ 4555.427562] Key type .llcrypt unregistered [ 4555.428731] Key type ._llcrypt unregistered [ 4569.515073] Key type ._llcrypt registered [ 4569.516244] Key type .llcrypt registered [ 4569.554508] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4569.956672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4569.974712] alg: No test for adler32 (adler32-zlib) [ 4570.833565] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4570.926790] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4572.512178] Key type lgssc registered [ 4572.910825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4575.846240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4575.854538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4576.942204] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4576.950695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4577.006287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.345509] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4581.185693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4581.220931] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4581.231450] Lustre: 225664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4581.331208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4582.787176] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4585.279069] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4587.178955] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4589.500924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4589.567299] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4589.639275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4591.551432] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4592.658555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 4594.052672] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4595.621612] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4604.223754] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4612.863931] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4621.457964] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4637.698240] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4637.701553] Lustre: Skipped 1 previous similar message [ 4658.352098] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4658.355192] Lustre: Skipped 2 previous similar messages [ 4691.054114] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4691.056272] Lustre: Skipped 4 previous similar messages [ 4756.450204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4756.451030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.454672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4756.457644] Lustre: Skipped 1 previous similar message [ 4759.569531] Lustre: server umount lustre-OST0000 complete [ 4761.569636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.570310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4761.573989] Lustre: Skipped 1 previous similar message [ 4761.575972] Lustre: Skipped 1 previous similar message [ 4766.688980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4766.691468] Lustre: Skipped 2 previous similar messages [ 4771.809039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4771.810926] Lustre: Skipped 1 previous similar message [ 4775.392119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4775.464202] Lustre: server umount lustre-MDT0000 complete [ 4776.855362] LustreError: 225656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365366 with bad export cookie 9141531999478187843 [ 4776.858135] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4776.861921] LustreError: 225656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4776.990203] Lustre: server umount lustre-MDT0001 complete [ 4780.822531] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4781.923438] Key type lgssc unregistered [ 4782.073473] LNet: 230454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4782.076967] LNetError: 230454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4782.085301] LNet: Removed LNI 192.168.201.156@tcp [ 4782.410439] Key type .llcrypt unregistered [ 4782.411877] Key type ._llcrypt unregistered [ 4791.983788] Key type ._llcrypt registered [ 4791.985558] Key type .llcrypt registered [ 4792.040766] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4792.385212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4792.427802] alg: No test for adler32 (adler32-zlib) [ 4793.290611] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4793.390564] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4794.984231] Key type lgssc registered [ 4795.444079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4798.408963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4798.417586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4799.566953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4801.088410] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4804.189044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4804.340307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4805.740891] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4808.393157] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4810.332660] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4812.616141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4812.734763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4814.780316] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4817.400088] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4826.087550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 4836.323397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4836.328031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.331311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4836.333285] Lustre: Skipped 1 previous similar message [ 4841.442672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4841.447102] Lustre: Skipped 2 previous similar messages [ 4842.372755] Lustre: server umount lustre-OST0000 complete [ 4846.562862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.568849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4846.572894] Lustre: Skipped 1 previous similar message [ 4846.579209] Lustre: Skipped 1 previous similar message [ 4851.681706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4851.689699] Lustre: Skipped 1 previous similar message [ 4851.819913] Lustre: server umount lustre-MDT0000 complete [ 4854.132710] LustreError: 231826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365443 with bad export cookie 3066321837977632015 [ 4854.139532] LustreError: 231826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4854.140798] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4854.403832] Lustre: server umount lustre-MDT0001 complete [ 4865.343337] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4867.349587] Key type lgssc unregistered [ 4867.532603] LNet: 235549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4867.536232] LNetError: 235549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4867.545983] LNet: Removed LNI 192.168.201.156@tcp [ 4868.066259] Key type .llcrypt unregistered [ 4868.068146] Key type ._llcrypt unregistered [ 4889.401828] Key type ._llcrypt registered [ 4889.407465] Key type .llcrypt registered [ 4889.556538] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4890.984620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4891.155295] alg: No test for adler32 (adler32-zlib) [ 4892.799395] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4893.441808] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4895.368194] Key type lgssc registered [ 4897.579446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5167.345413] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5169.186748] Key type lgssc unregistered [ 5169.407660] LNet: 237573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5169.417422] LNetError: 237573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5169.433758] LNet: Removed LNI 192.168.201.156@tcp [ 5170.026563] Key type .llcrypt unregistered [ 5170.033162] Key type ._llcrypt unregistered [ 5187.995397] Key type ._llcrypt registered [ 5187.997501] Key type .llcrypt registered [ 5188.081254] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 5198.938801] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5199.745576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5199.775267] alg: No test for adler32 (adler32-zlib) [ 5200.784488] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 5200.946885] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 5202.584180] Key type lgssc registered [ 5203.488737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5221.713653] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4906 sec ========== 12:16:50 (1788365810) [ 5222.740226] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:16:51 (1788365811) === [ 5224.925552] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:16:53 (1788365813) === [ 5249.923260] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5251.562340] Key type lgssc unregistered [ 5251.721546] LNet: 242784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5251.726332] LNetError: 242784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5251.738705] LNet: Removed LNI 192.168.201.156@tcp [ 5252.262423] Key type .llcrypt unregistered [ 5252.264305] Key type ._llcrypt unregistered