[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 452423195 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003133] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005029] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009027] pid_max: default: 32768 minimum: 301 [ 0.010153] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.014020] *** VALIDATE selinux *** [ 0.022531] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027155] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029031] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031136] *** VALIDATE tmpfs *** [ 0.032441] *** VALIDATE proc *** [ 0.033275] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036145] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.042661] debug: unmapping init [mem 0xffffffffa7459000-0xffffffffa7460fff] [ 0.045210] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046713] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049018] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054323] rcu: Hierarchical SRCU implementation. [ 0.056511] smp: Bringing up secondary CPUs ... [ 0.057584] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.061517] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141344] node 0 deferred pages initialised in 74ms [ 0.144110] devtmpfs: initialized [ 0.146222] x86/mm: Memory block size: 128MB [ 0.148772] gcov: version magic: 0x41383552 [ 0.150166] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152269] pinctrl core: initialized pinctrl subsystem [ 0.153230] [ 0.153780] ************************************************************* [ 0.154013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155017] ** ** [ 0.156017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157013] ** ** [ 0.158016] ** This means that this kernel is built to expose internal ** [ 0.159014] ** IOMMU data structures, which may compromise security on ** [ 0.160012] ** your system. ** [ 0.161010] ** ** [ 0.162016] ** If you see this message and you are not debugging the ** [ 0.163014] ** kernel, report this immediately to your vendor! ** [ 0.164016] ** ** [ 0.165015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166014] ************************************************************* [ 0.167611] NET: Registered protocol family 16 [ 0.168408] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171509] cpuidle: using governor menu [ 0.174033] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177489] PCI: Using configuration type 1 for base access [ 0.179132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.191161] cryptd: max_cpu_qlen set to 1000 [ 0.193096] ACPI: Added _OSI(Module Device) [ 0.194005] ACPI: Added _OSI(Processor Device) [ 0.194961] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199049] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201468] ACPI: Interpreter enabled [ 0.203015] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203988] ACPI: Using IOAPIC for interrupt routing [ 0.204094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205415] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.213830] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.215019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.218281] acpiphp: Slot [2] registered [ 0.219066] acpiphp: Slot [5] registered [ 0.219878] acpiphp: Slot [6] registered [ 0.220080] acpiphp: Slot [7] registered [ 0.221097] acpiphp: Slot [8] registered [ 0.222115] acpiphp: Slot [9] registered [ 0.223127] acpiphp: Slot [10] registered [ 0.224138] acpiphp: Slot [3] registered [ 0.225142] acpiphp: Slot [4] registered [ 0.226095] acpiphp: Slot [11] registered [ 0.227104] acpiphp: Slot [12] registered [ 0.228096] acpiphp: Slot [13] registered [ 0.229091] acpiphp: Slot [14] registered [ 0.230080] acpiphp: Slot [15] registered [ 0.231059] acpiphp: Slot [16] registered [ 0.232116] acpiphp: Slot [17] registered [ 0.233110] acpiphp: Slot [18] registered [ 0.234108] acpiphp: Slot [19] registered [ 0.235101] acpiphp: Slot [20] registered [ 0.236110] acpiphp: Slot [21] registered [ 0.237094] acpiphp: Slot [22] registered [ 0.238128] acpiphp: Slot [23] registered [ 0.239097] acpiphp: Slot [24] registered [ 0.240095] acpiphp: Slot [25] registered [ 0.241103] acpiphp: Slot [26] registered [ 0.242132] acpiphp: Slot [27] registered [ 0.243085] acpiphp: Slot [28] registered [ 0.244096] acpiphp: Slot [29] registered [ 0.245111] acpiphp: Slot [30] registered [ 0.246095] acpiphp: Slot [31] registered [ 0.247099] PCI host bridge to bus 0000:00 [ 0.248020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.249023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.250022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.251025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.252023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.253026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.254188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.255997] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.258284] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.264016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.267058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.268014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.269021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.270020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.271584] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.272781] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.273041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.274789] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.277016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.285927] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.287017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.289961] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.292017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.295018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.302016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.307799] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.311017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.314017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.321026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.327676] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.330016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.333016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.339020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.344906] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.347021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.350018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.358015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.363045] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.366019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.369026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.376017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.381820] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.384018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.387020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.394021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.401815] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.402376] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.403318] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.404247] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.405236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.408133] iommu: Default domain type: Passthrough [ 0.409418] SCSI subsystem initialized [ 0.410122] ACPI: bus type USB registered [ 0.411086] usbcore: registered new interface driver usbfs [ 0.412058] usbcore: registered new interface driver hub [ 0.413069] usbcore: registered new device driver usb [ 0.414141] pps_core: LinuxPPS API ver. 1 registered [ 0.415011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.416069] PTP clock support registered [ 0.417162] EDAC MC: Ver: 3.0.0 [ 0.419120] PCI: Using ACPI for IRQ routing [ 0.420660] NetLabel: Initializing [ 0.421009] NetLabel: domain hash size = 128 [ 0.421977] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.422064] NetLabel: unlabeled traffic allowed by default [ 0.423150] vgaarb: loaded [ 0.424281] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.425011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.430000] clocksource: Switched to clocksource kvm-clock [ 0.531671] VFS: Disk quotas dquot_6.6.0 [ 0.532782] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.534815] *** VALIDATE ramfs *** [ 0.535971] *** VALIDATE hugetlbfs *** [ 0.537381] pnp: PnP ACPI init [ 0.539625] pnp: PnP ACPI: found 6 devices [ 0.561787] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.565147] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.567072] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.569166] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.571561] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.573956] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.576699] NET: Registered protocol family 2 [ 0.579086] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.583360] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.585863] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.590886] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.593897] TCP: Hash tables configured (established 65536 bind 65536) [ 0.596422] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.599242] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.602070] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.604600] NET: Registered protocol family 1 [ 0.606918] RPC: Registered named UNIX socket transport module. [ 0.608930] RPC: Registered udp transport module. [ 0.610592] RPC: Registered tcp transport module. [ 0.612346] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.614518] NET: Registered protocol family 44 [ 0.616120] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.618284] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.620190] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.622649] PCI: CLS 0 bytes, default 64 [ 0.624351] Unpacking initramfs... [ 2.008615] debug: unmapping init [mem 0xffff8c99fcc54000-0xffff8c99fffbffff] [ 2.012473] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.014837] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.017756] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.498125] Initialise system trusted keyrings [ 2.499913] Key type blacklist registered [ 2.501885] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.511813] zbud: loaded [ 2.515196] *** VALIDATE nfs *** [ 2.516598] *** VALIDATE nfs4 *** [ 2.518229] pstore: using deflate compression [ 2.522225] Platform Keyring initialized [ 2.615214] NET: Registered protocol family 38 [ 2.616620] Key type asymmetric registered [ 2.617667] Asymmetric key parser 'x509' registered [ 2.618835] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.620727] io scheduler mq-deadline registered [ 2.622281] io scheduler kyber registered [ 2.623674] io scheduler bfq registered [ 2.625376] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.627805] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.629878] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.631883] ACPI: Power Button [PWRF] [ 2.635741] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.640602] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.650816] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.656195] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.668718] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.694699] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.719910] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.723939] Non-volatile memory driver v1.3 [ 2.725181] Linux agpgart interface v0.103 [ 2.750372] virtio_blk virtio1: [vda] 146712 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.753247] vda: detected capacity change from 0 to 75116544 [ 2.769731] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.772870] vdb: detected capacity change from 0 to 1073741824 [ 2.789279] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.791663] vdc: detected capacity change from 0 to 2621440000 [ 2.809635] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.812143] vdd: detected capacity change from 0 to 2621440000 [ 2.830245] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.833873] vde: detected capacity change from 0 to 4294967296 [ 2.859897] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.863140] vdf: detected capacity change from 0 to 4294967296 [ 2.877092] libphy: Fixed MDIO Bus: probed [ 2.886362] usbcore: registered new interface driver usbserial_generic [ 2.888932] usbserial: USB Serial support registered for generic [ 2.892896] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.896856] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.898717] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.901170] mousedev: PS/2 mouse device common for all mice [ 2.904616] rtc_cmos 00:05: RTC can wake from S4 [ 2.907231] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.912185] rtc_cmos 00:05: registered as rtc0 [ 2.914140] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.914671] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.916955] intel_pstate: CPU model not supported [ 2.921732] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.926218] hid: raw HID events driver (C) Jiri Kosina [ 2.927827] usbcore: registered new interface driver usbhid [ 2.929407] usbhid: USB HID core driver [ 2.930730] drop_monitor: Initializing network drop monitor service [ 2.932554] Initializing XFRM netlink socket [ 2.934359] NET: Registered protocol family 10 [ 2.936936] Segment Routing with IPv6 [ 2.937937] NET: Registered protocol family 17 [ 2.939548] mpls_gso: MPLS GSO support [ 2.945730] RAS: Correctable Errors collector initialized. [ 2.947392] AVX version of gcm_enc/dec engaged. [ 2.948826] AES CTR mode by8 optimization enabled [ 3.010951] sched_clock: Marking stable (3010927505, 0)->(4057376377, -1046448872) [ 3.014175] registered taskstats version 1 [ 3.015727] Loading compiled-in X.509 certificates [ 3.017697] zswap: loaded using pool lzo/zbud [ 3.039971] Key type big_key registered [ 3.052030] Key type encrypted registered [ 3.053161] ima: No TPM chip found, activating TPM-bypass! [ 3.054453] ima: Allocated hash algorithm: sha1 [ 3.055466] ima: No architecture policies found [ 3.056446] evm: Initialising EVM extended attributes: [ 3.057478] evm: security.selinux [ 3.058327] evm: security.ima [ 3.058894] evm: security.capability [ 3.059757] evm: HMAC attrs: 0x1 [ 3.062082] rtc_cmos 00:05: setting system clock to 2026-09-08 05:06:46 UTC (1788844006) [ 3.066795] debug: unmapping init [mem 0xffffffffa8403000-0xffffffffa85fffff] [ 3.069594] debug: unmapping init [mem 0xffffffffa7182000-0xffffffffa7458fff] [ 3.078336] Write protecting the kernel read-only data: 28672k [ 3.081829] debug: unmapping init [mem 0xffffffffa5803000-0xffffffffa59fffff] [ 3.084490] debug: unmapping init [mem 0xffffffffa6114000-0xffffffffa61fffff] [ 3.117145] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.125735] systemd[1]: Detected virtualization kvm. [ 3.127627] systemd[1]: Detected architecture x86-64. [ 3.129877] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.158822] systemd[1]: No hostname configured. [ 3.160561] systemd[1]: Set hostname to . [ 3.162762] random: systemd: uninitialized urandom read (16 bytes read) [ 3.164826] systemd[1]: Initializing machine ID from random generator. [ 3.286692] random: systemd: uninitialized urandom read (16 bytes read) [ 3.289114] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.293154] random: systemd: uninitialized urandom read (16 bytes read) [ 3.295422] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.300157] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.843460] device-mapper: uevent: version 1.0.3 [ 3.845842] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [[ 4.448612] random: fast init done  OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.538112] virtio_net virtio0 ens2: renamed from eth0 [ 4.573853] scsi host0: ata_piix [ 4.591847] scsi host1: ata_piix [ 4.630844] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.632605] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.840159] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.567796] random: crng init done [ 9.569123] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.846715] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.931889] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.199262] SELinux: Disabled at runtime. [ 11.260223] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.269051] systemd[1]: Detected virtualization kvm. [ 11.271066] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.771335] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.774966] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.786830] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.791237] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.794955] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.807502] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.814981] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 11.932050] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.241722] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.460245] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.515269] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.606803] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.616437] EDAC sbridge: Ver: 1.1.2 [ 14.239119] Key type dns_resolver registered [ 14.550988] NFS: Registering the id_resolver key type [ 14.553428] Key type id_resolver registered [ 14.554941] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg253-server login: [ 58.580792] hrtimer: interrupt took 4173736 ns [ 78.126693] libcfs: loading out-of-tree module taints kernel. [ 78.186746] Key type ._llcrypt registered [ 78.193569] Key type .llcrypt registered [ 78.328929] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 98.537681] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 100.680803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 100.712721] alg: No test for adler32 (adler32-zlib) [ 102.355727] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 103.627571] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 105.327178] Key type lgssc registered [ 107.614929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.545420] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 170.313811] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 182.575648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 182.600950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.909645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 183.928868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 183.983363] Lustre: lustre-MDT0000: new disk, initializing [ 184.042871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.058588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.091991] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.273674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.375396] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.400665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 202.407644] Lustre: Skipped 1 previous similar message [ 202.523533] Lustre: lustre-MDT0001: new disk, initializing [ 202.636117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.673926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 202.685360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 206.723894] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.152312] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.396770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.637830] Lustre: lustre-OST0000: new disk, initializing [ 220.641065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.645631] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.692443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.225796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 222.238392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 222.335191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.501968] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.786899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.902798] Lustre: lustre-OST0001: new disk, initializing [ 240.905713] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.910144] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.971995] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 247.266956] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.869703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 247.878510] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 247.974449] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 259.866604] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.558841] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.304751] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.407336] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing yml_node [ 287.498960] Lustre: DEBUG MARKER: Client: 2.17.57.111 [ 290.432826] Lustre: DEBUG MARKER: MDS: 2.17.57.111 [ 293.092543] Lustre: DEBUG MARKER: OSS: 2.17.57.111 [ 295.286578] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 01:11:37 EDT 2026 [ 311.527837] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 313.536504] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.219742] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.229328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.250120] Lustre: Skipped 3 previous similar messages [ 330.280979] Lustre: Skipped 3 previous similar messages [ 335.332548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.339835] Lustre: Skipped 3 previous similar messages [ 335.623213] Lustre: server umount lustre-MDT0000 complete [ 339.427551] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844342 with bad export cookie 8540801304032901615 [ 339.436566] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 339.440162] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 340.449107] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.471425] Lustre: Skipped 1 previous similar message [ 340.475362] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.485691] Lustre: Skipped 1 previous similar message [ 344.480457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.487857] Lustre: Skipped 1 previous similar message [ 345.916329] Lustre: server umount lustre-MDT0001 complete [ 350.024311] Lustre: server umount lustre-OST0000 complete [ 354.002482] Lustre: server umount lustre-OST0001 complete [ 361.112351] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 371.352936] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 407.842183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.055961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 408.084321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 408.157335] Lustre: lustre-MDT0000: new disk, initializing [ 408.239220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.264501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 412.447420] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 422.638872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.811546] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 422.912179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 422.916041] Lustre: Skipped 1 previous similar message [ 423.011654] Lustre: lustre-MDT0001: new disk, initializing [ 423.163783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 423.202214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 423.233808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 427.644737] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 437.252722] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 442.693879] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 449.388724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 449.718314] Lustre: lustre-OST0000: new disk, initializing [ 449.726583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 449.742990] Lustre: 17043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 449.885269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.850671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 450.871265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 450.980607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 456.619816] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 465.631168] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 469.956560] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 470.223432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 474.427706] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 474.597863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 476.653605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.653911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 476.667468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 476.671927] LustreError: Skipped 1 previous similar message [ 476.675807] Lustre: Skipped 1 previous similar message [ 482.098875] Lustre: server umount lustre-OST0000 complete [ 486.884306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.918646] Lustre: Skipped 1 previous similar message [ 486.923915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.942947] Lustre: Skipped 3 previous similar messages [ 491.923499] Lustre: server umount lustre-MDT0000 complete [ 492.002935] LustreError: 15923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.031578] LustreError: 15923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 496.144373] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844499 with bad export cookie 8540801304032903141 [ 496.158631] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 496.160765] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 496.825812] Lustre: server umount lustre-MDT0001 complete [ 501.595297] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:15:03 (1788844503) [ 508.607672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.003534] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.075818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.993458] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 514.528474] LustreError: 19307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.628911] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.544055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.232723] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 534.087710] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 539.188606] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 546.604833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 553.011282] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 560.589484] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 569.824671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.828970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 569.835214] Lustre: Skipped 1 previous similar message [ 569.839388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 569.852179] Lustre: Skipped 1 previous similar message [ 576.163763] Lustre: server umount lustre-OST0000 complete [ 584.161423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 584.175668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.186982] Lustre: Skipped 1 previous similar message [ 586.709567] Lustre: server umount lustre-MDT0000 complete [ 590.332158] LustreError: 19302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.350432] LustreError: 19302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 590.509716] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844593 with bad export cookie 8540801304032904030 [ 590.513059] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 590.522943] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 590.880982] Lustre: server umount lustre-MDT0001 complete [ 600.675317] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 603.711979] Key type lgssc unregistered [ 603.971741] LNet: 22836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 603.981026] LNetError: 22836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 603.994869] LNet: Removed LNI 192.168.202.153@tcp [ 604.957379] Key type .llcrypt unregistered [ 604.960056] Key type ._llcrypt unregistered [ 634.080705] Key type ._llcrypt registered [ 634.083432] Key type .llcrypt registered [ 634.228235] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 635.991870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 636.046568] alg: No test for adler32 (adler32-zlib) [ 637.109770] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 637.519092] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 639.231353] Key type lgssc registered [ 640.888646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 650.132630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 650.183780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.939049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.337172] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 662.510395] LustreError: 24246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.548761] LustreError: 24246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 663.591637] LustreError: 24247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.642235] LustreError: 24246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 669.451568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.090422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 676.162677] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 685.829597] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 694.377928] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 701.847431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 702.229890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 708.785993] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 716.741968] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 723.765493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 734.179856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 734.184470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.209955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 734.218784] Lustre: Skipped 1 previous similar message [ 739.314071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 739.325125] Lustre: Skipped 1 previous similar message [ 740.238826] Lustre: server umount lustre-OST0000 complete [ 744.428856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 744.446763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 744.467915] Lustre: Skipped 2 previous similar messages [ 744.487320] Lustre: Skipped 1 previous similar message [ 749.536339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 749.543982] Lustre: Skipped 1 previous similar message [ 750.466603] Lustre: server umount lustre-MDT0000 complete [ 754.599161] LustreError: 24227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844758 with bad export cookie 11004633972167048035 [ 754.601220] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 754.607556] LustreError: 24227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 754.964446] Lustre: server umount lustre-MDT0001 complete [ 765.697743] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 768.612182] Key type lgssc unregistered [ 769.057574] LNet: 28227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.066099] LNetError: 28227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.079322] LNet: Removed LNI 192.168.202.153@tcp [ 769.821922] Key type .llcrypt unregistered [ 769.826088] Key type ._llcrypt unregistered [ 796.438437] Key type ._llcrypt registered [ 796.440377] Key type .llcrypt registered [ 796.669994] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 798.345561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 798.364283] alg: No test for adler32 (adler32-zlib) [ 799.729536] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 800.140053] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 801.961113] Key type lgssc registered [ 803.720429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 812.811925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 812.844537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 814.376969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.533351] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 823.787958] LustreError: 29639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 823.822600] LustreError: 29639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 828.634945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.907634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 833.448868] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 841.714488] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 847.807560] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 861.603714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 862.002931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 868.823289] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 876.070400] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 884.560489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 897.312469] Lustre: server umount lustre-OST0000 complete [ 905.202602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.203483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 905.212614] Lustre: Skipped 1 previous similar message [ 910.305409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 910.326942] Lustre: Skipped 1 previous similar message [ 912.724988] Lustre: server umount lustre-MDT0000 complete [ 915.426392] LustreError: 29634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.471633] LustreError: 29634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 916.454860] Lustre: 28949:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844903/real 1788844903] req@00000000e7bba1ae x1875739728181888/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788844919 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 916.489831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 916.500230] LustreError: 29639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 916.514134] LustreError: 29639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 918.766539] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844922 with bad export cookie 6250644200654071606 [ 918.789265] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 918.797695] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 919.199275] Lustre: server umount lustre-MDT0001 complete [ 932.171642] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 935.380630] Key type lgssc unregistered [ 935.792919] LNet: 33538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.826667] LNetError: 33538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.859605] LNet: Removed LNI 192.168.202.153@tcp [ 936.760752] Key type .llcrypt unregistered [ 936.763474] Key type ._llcrypt unregistered [ 963.188371] Key type ._llcrypt registered [ 963.192571] Key type .llcrypt registered [ 963.368871] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 964.915980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 964.947382] alg: No test for adler32 (adler32-zlib) [ 966.213317] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 966.651438] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 968.394195] Key type lgssc registered [ 969.789749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 978.740598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 978.764644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.494264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.203957] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 989.665474] LustreError: 34959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.690675] LustreError: 34959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 994.788698] LustreError: 34958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.077638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.512835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1001.756233] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1010.606655] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1016.404442] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1022.657836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1022.995556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.083942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1030.684357] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1041.030962] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1051.616674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1051.625887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.634426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1051.673036] Lustre: Skipped 1 previous similar message [ 1056.738883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1056.754099] Lustre: Skipped 1 previous similar message [ 1057.336745] Lustre: server umount lustre-OST0000 complete [ 1061.856351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.861219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1061.867092] Lustre: Skipped 1 previous similar message [ 1061.875027] Lustre: Skipped 1 previous similar message [ 1066.978062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1066.995129] Lustre: Skipped 2 previous similar messages [ 1068.171190] Lustre: server umount lustre-MDT0000 complete [ 1072.096994] LustreError: 34958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.143523] LustreError: 34958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1072.263900] LustreError: 34939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845075 with bad export cookie 8402473871701055756 [ 1072.267421] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1072.291620] LustreError: 34939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1072.904701] Lustre: server umount lustre-MDT0001 complete [ 1085.137753] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1088.143239] Key type lgssc unregistered [ 1088.369142] LNet: 38496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.383497] LNetError: 38496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.462238] LNet: Removed LNI 192.168.202.153@tcp [ 1090.320839] Key type .llcrypt unregistered [ 1090.322846] Key type ._llcrypt unregistered [ 1118.953507] Key type ._llcrypt registered [ 1118.956495] Key type .llcrypt registered [ 1119.057301] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1120.625033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1120.650336] alg: No test for adler32 (adler32-zlib) [ 1121.821312] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1122.115871] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1123.959183] Key type lgssc registered [ 1125.411593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1133.831132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1133.859774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.329253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.899524] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1144.811259] LustreError: 39910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.830266] LustreError: 39910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1149.938727] LustreError: 39909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.051124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.532602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1155.523801] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1163.411472] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1170.284891] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1177.024954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1177.444781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1180.566675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1183.671339] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1190.093743] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1200.099059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.108464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1200.109168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1203.105184] Lustre: lustre-OST0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 1203.113823] Lustre: Skipped 1 previous similar message [ 1205.222277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.224704] Lustre: Skipped 1 previous similar message [ 1206.181039] Lustre: server umount lustre-OST0000 complete [ 1227.242760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1227.257635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.270238] Lustre: Skipped 1 previous similar message [ 1227.278604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1230.400376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.210264] Lustre: server umount lustre-MDT0000 complete [ 1235.424987] LustreError: 39909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.451805] LustreError: 39909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1235.732227] LustreError: 39888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845239 with bad export cookie 18316535216919668865 [ 1235.746973] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.749547] LustreError: 39888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1250.271981] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1250.779715] Lustre: server umount lustre-MDT0001 complete [ 1264.660938] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1268.388796] Key type lgssc unregistered [ 1268.894286] LNet: 43617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.923462] LNetError: 43617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.941312] LNet: Removed LNI 192.168.202.153@tcp [ 1270.455304] Key type .llcrypt unregistered [ 1270.462633] Key type ._llcrypt unregistered [ 1298.682763] Key type ._llcrypt registered [ 1298.685992] Key type .llcrypt registered [ 1298.829814] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1300.191940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1300.295668] alg: No test for adler32 (adler32-zlib) [ 1301.371600] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1301.836861] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1303.671209] Key type lgssc registered [ 1304.919962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1315.706884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1315.723424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1317.412343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.297677] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1326.585100] LustreError: 45029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.614082] LustreError: 45029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1331.688931] LustreError: 45030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.601571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.154043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1338.720461] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1347.132652] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1353.607354] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1360.589853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1360.905848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.919778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1367.565568] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1375.483263] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1388.521160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.529383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1388.540522] Lustre: Skipped 1 previous similar message [ 1388.560388] Lustre: Skipped 3 previous similar messages [ 1393.395021] Lustre: server umount lustre-MDT0000 complete [ 1393.547279] LustreError: 45026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.466713] LustreError: 45009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845401 with bad export cookie 17430461409538165364 [ 1398.477370] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1398.497850] LustreError: 45009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1398.651781] LustreError: 45024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.696378] LustreError: 45024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1398.737340] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.53@tcp (stopping) [ 1398.771457] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.798194] Lustre: Skipped 1 previous similar message [ 1403.774435] LustreError: 45024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.776216] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.53@tcp (stopping) [ 1403.817301] LustreError: 45024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1403.850402] Lustre: Skipped 1 previous similar message [ 1405.190832] Lustre: server umount lustre-MDT0001 complete [ 1420.775767] Lustre: server umount lustre-OST0000 complete [ 1437.937708] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1440.728372] Key type lgssc unregistered [ 1440.898705] LNet: 48889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.902572] LNetError: 48889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.911814] LNet: Removed LNI 192.168.202.153@tcp [ 1441.458912] Key type .llcrypt unregistered [ 1441.461169] Key type ._llcrypt unregistered [ 1469.154708] Key type ._llcrypt registered [ 1469.158436] Key type .llcrypt registered [ 1469.284443] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1470.879862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1470.916390] alg: No test for adler32 (adler32-zlib) [ 1472.128786] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1472.433877] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1474.137754] Key type lgssc registered [ 1475.211291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1482.843410] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1482.860682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1498.463763] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1503.584531] LustreError: 50272:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 1530.271598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.906864] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1544.922379] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1709.622861] Lustre: server umount lustre-OST0000 complete [ 1728.610826] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1731.977768] Key type lgssc unregistered [ 1732.414072] LNet: 51946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.419921] LNetError: 51946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.447688] LNet: Removed LNI 192.168.202.153@tcp [ 1733.570866] Key type .llcrypt unregistered [ 1733.580539] Key type ._llcrypt unregistered [ 1759.964936] Key type ._llcrypt registered [ 1759.969289] Key type .llcrypt registered [ 1760.082426] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1761.510963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1761.540257] alg: No test for adler32 (adler32-zlib) [ 1762.698296] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1763.170513] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1764.888105] Key type lgssc registered [ 1766.224825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1775.523609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1775.551632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.980187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.149256] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1787.370603] LustreError: 53357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.395594] LustreError: 53357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1790.561368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.003944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1796.241827] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1804.048327] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1809.794407] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1816.447530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1816.753496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.901397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1822.193830] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1829.693258] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1832.418729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1832.419483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.448831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1836.585125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1838.453315] Lustre: server umount lustre-OST0000 complete [ 1842.145860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1842.148872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.165221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1842.171199] Lustre: Skipped 1 previous similar message [ 1846.762911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.768784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1846.784690] Lustre: Skipped 1 previous similar message [ 1848.516367] Lustre: server umount lustre-MDT0000 complete [ 1851.879461] LustreError: 53356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.902423] LustreError: 53356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1852.511925] LustreError: 53336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845855 with bad export cookie 10848385754182503821 [ 1852.530981] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1852.956282] Lustre: server umount lustre-MDT0001 complete [ 1868.207457] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1871.508918] Key type lgssc unregistered [ 1871.874321] LNet: 56886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.886522] LNetError: 56886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.904828] LNet: Removed LNI 192.168.202.153@tcp [ 1872.916676] Key type .llcrypt unregistered [ 1872.919256] Key type ._llcrypt unregistered [ 1898.922844] Key type ._llcrypt registered [ 1898.925387] Key type .llcrypt registered [ 1899.065518] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1900.629909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1900.671266] alg: No test for adler32 (adler32-zlib) [ 1901.678724] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 1901.951792] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1903.687311] Key type lgssc registered [ 1905.006979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1913.702481] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1913.730917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1929.375632] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1934.495633] LustreError: 58269:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 1961.119260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.172871] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1974.055433] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1981.709744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.424377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.514086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1987.552879] LustreError: 58294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.577288] LustreError: 58294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1987.940996] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1992.687393] LustreError: 58295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1995.751397] LustreError: 59307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1995.789804] LustreError: 59307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1999.654505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.222331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2006.309257] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2013.969122] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2021.161354] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2023.392516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2023.402819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.421573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2025.960333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2025.971473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2025.974601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.021861] Lustre: Skipped 1 previous similar message [ 2029.547736] Lustre: server umount lustre-OST0000 complete [ 2056.687213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.690745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2056.700548] Lustre: Skipped 1 previous similar message [ 2056.717178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2056.729118] Lustre: Skipped 1 previous similar message [ 2061.793130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2061.813214] Lustre: Skipped 1 previous similar message [ 2065.893651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2065.913127] Lustre: Skipped 1 previous similar message [ 2066.399389] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2066.676381] Lustre: server umount lustre-MDT0000 complete [ 2070.744065] LustreError: 58277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846074 with bad export cookie 11684026089071680147 [ 2070.748828] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2070.758762] LustreError: 58277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2085.343355] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2085.808597] Lustre: server umount lustre-MDT0001 complete [ 2098.507864] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2101.665199] Key type lgssc unregistered [ 2101.972837] LNet: 62018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.983388] LNetError: 62018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.010094] LNet: Removed LNI 192.168.202.153@tcp [ 2102.868991] Key type .llcrypt unregistered [ 2102.870752] Key type ._llcrypt unregistered [ 2128.211603] Key type ._llcrypt registered [ 2128.214702] Key type .llcrypt registered [ 2128.414128] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2129.805048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2129.841605] alg: No test for adler32 (adler32-zlib) [ 2130.948476] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2131.354924] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2133.071259] Key type lgssc registered [ 2134.528444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2143.723967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2143.770071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.429928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.198430] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2154.991628] LustreError: 63428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.015324] LustreError: 63428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2158.222299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.523800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2162.583366] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2169.793945] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2175.943760] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2183.089924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2183.460181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.614850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2190.050533] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2197.986854] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2215.393248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2215.400458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.414823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2218.464755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2218.466048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2218.480744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.925984] Lustre: server umount lustre-OST0000 complete [ 2225.120407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2225.134831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.150610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2225.156669] Lustre: Skipped 1 previous similar message [ 2228.706242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.708048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.610971] Lustre: server umount lustre-MDT0000 complete [ 2233.823976] LustreError: 63428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.845539] LustreError: 63428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2234.359820] LustreError: 63409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846237 with bad export cookie 9552475363439810910 [ 2234.363361] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2234.368489] LustreError: 63409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2234.623910] Lustre: server umount lustre-MDT0001 complete [ 2244.904750] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2248.019871] Key type lgssc unregistered [ 2248.424444] LNet: 66981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.432535] LNetError: 66981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.450346] LNet: Removed LNI 192.168.202.153@tcp [ 2249.395476] Key type .llcrypt unregistered [ 2249.398546] Key type ._llcrypt unregistered [ 2273.821961] Key type ._llcrypt registered [ 2273.825797] Key type .llcrypt registered [ 2273.945796] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2274.587618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2274.617681] alg: No test for adler32 (adler32-zlib) [ 2275.831864] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2276.028971] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2277.711172] Key type lgssc registered [ 2278.666091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2288.876350] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 01:44:50 (1788846290) [ 2297.299200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2297.364090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.787294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.192544] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2309.090534] LustreError: 68660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.099278] LustreError: 68660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2312.373038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.782469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2317.690754] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2325.449488] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2332.018666] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2339.415176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2340.069501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.115866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2345.789789] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2354.394926] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2363.337957] Lustre: Failing over lustre-MDT0000 [ 2363.875026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.878905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2363.888860] Lustre: Skipped 1 previous similar message [ 2363.895372] LustreError: 68661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.937133] LustreError: 68661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2364.131305] Lustre: server umount lustre-MDT0000 complete [ 2366.390845] LustreError: 68656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.008287] LustreError: 70202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.031571] LustreError: 70202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2374.119565] LustreError: 68660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.147121] LustreError: 68660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2375.164048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.305098] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2375.431737] Lustre: *** cfs_fail_loc=135, val=0*** [ 2375.554796] LustreError: 71748:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2375.558491] LustreError: 71748:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.153@tcp: cfg command failed: rc = -2 [ 2375.566441] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2375.566441] [ 2375.576727] LustreError: MGC192.168.202.153@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2375.592656] LustreError: 71735:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2375.613581] LustreError: 71735:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2375.626748] LustreError: 71735:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2375.636558] LustreError: 71735:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2375.681403] Lustre: server umount lustre-MDT0000 complete [ 2375.684623] LustreError: 71735:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2382.791397] LustreError: 68656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.807441] LustreError: 68656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2382.898075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.032369] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.044425] LustreError: 68652:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2383.051621] LustreError: 72095:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.153@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000001e3f1a01) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2383.072386] LustreError: 67727:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d09aff7e x1875741275880576/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2383.429248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.497366] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2386.827623] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2388.008041] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2388.464121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2388.496620] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2388.558625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2398.707639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.721071] Lustre: Skipped 1 previous similar message [ 2398.733422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2401.478515] Lustre: server umount lustre-OST0000 complete [ 2408.929435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.936411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2408.947871] Lustre: Skipped 1 previous similar message [ 2408.954133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2408.954140] Lustre: Skipped 1 previous similar message [ 2411.538415] Lustre: server umount lustre-MDT0000 complete [ 2414.055920] LustreError: 68661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.064187] LustreError: 68661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2416.796271] LustreError: 68643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846420 with bad export cookie 13157412995872219710 [ 2416.819360] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2417.361417] Lustre: server umount lustre-MDT0001 complete [ 2431.264630] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2433.702639] Key type lgssc unregistered [ 2433.958837] LNet: 73530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.973685] LNetError: 73530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2434.008872] LNet: Removed LNI 192.168.202.153@tcp [ 2434.929920] Key type .llcrypt unregistered [ 2434.931184] Key type ._llcrypt unregistered [ 2456.838731] Key type ._llcrypt registered [ 2456.844695] Key type .llcrypt registered [ 2457.083979] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2458.640709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2458.675423] alg: No test for adler32 (adler32-zlib) [ 2459.831503] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2459.983511] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2461.615251] Key type lgssc registered [ 2462.716846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2472.951962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2472.971581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.701458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.756459] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2485.239088] LustreError: 74918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.258763] LustreError: 74918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2490.353990] LustreError: 74919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.303464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.577335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2496.084554] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2504.381172] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2510.598927] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2518.460352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.838290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.866884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2526.299119] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2535.278445] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2542.409850] Lustre: Failing over lustre-MDT0000 [ 2542.829463] Lustre: server umount lustre-MDT0000 complete [ 2543.073649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2543.082475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.099358] LustreError: 75647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.664403] LustreError: 74919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.670936] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.696438] LustreError: 74919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2551.784138] LustreError: 76056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.799028] LustreError: 76056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2552.781972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.896643] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.002322] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2553.088095] LustreError: 78006:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2553.092146] LustreError: 78006:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.153@tcp: cfg command failed: rc = -9 [ 2553.098306] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2553.098306] [ 2553.106278] LustreError: MGC192.168.202.153@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2553.118958] LustreError: 77994:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2553.124302] LustreError: 77994:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 2553.133787] LustreError: 77994:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2553.139049] LustreError: 77994:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2553.159627] Lustre: server umount lustre-MDT0000 complete [ 2553.163384] LustreError: 77994:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2559.920932] LustreError: 76056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.933312] LustreError: 76056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2560.001862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.090470] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2560.362525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.411723] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2564.993748] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2565.499093] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2565.625846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2565.660427] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2565.723033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2575.854105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2575.866605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2575.872452] Lustre: Skipped 2 previous similar messages [ 2575.892100] Lustre: Skipped 1 previous similar message [ 2580.975991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2580.990132] Lustre: Skipped 1 previous similar message [ 2581.423497] Lustre: server umount lustre-OST0000 complete [ 2591.201199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2591.203484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.227954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2591.254304] Lustre: Skipped 1 previous similar message [ 2593.131542] Lustre: server umount lustre-MDT0000 complete [ 2596.322867] LustreError: 75647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.343168] LustreError: 75647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2598.105208] LustreError: 74900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846601 with bad export cookie 1692218462760307241 [ 2598.124639] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2598.727593] Lustre: server umount lustre-MDT0001 complete [ 2611.646458] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2615.043341] Key type lgssc unregistered [ 2615.419955] LNet: 79786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2615.428235] LNetError: 79786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2615.440992] LNet: Removed LNI 192.168.202.153@tcp [ 2616.571565] Key type .llcrypt unregistered [ 2616.573251] Key type ._llcrypt unregistered [ 2630.754533] Key type ._llcrypt registered [ 2630.759656] Key type .llcrypt registered [ 2630.893651] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2631.282493] Key type .llcrypt unregistered [ 2631.287092] Key type ._llcrypt unregistered [ 2655.719660] Key type ._llcrypt registered [ 2655.726667] Key type .llcrypt registered [ 2655.842137] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2656.840092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2656.938478] alg: No test for adler32 (adler32-zlib) [ 2658.329143] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2658.690627] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2660.375184] Key type lgssc registered [ 2662.034979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2670.995802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2671.030974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.725118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.744055] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2682.867308] LustreError: 81685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.890721] LustreError: 81685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2687.899989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.970740] LustreError: 81686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.554144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2694.200337] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2702.809724] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2709.056204] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2715.640356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2715.920038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2718.001136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2721.874288] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2730.353961] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2748.899091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.917197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2748.919045] Lustre: Skipped 1 previous similar message [ 2748.929662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2750.311090] Lustre: server umount lustre-OST0000 complete [ 2754.017493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.023556] Lustre: Skipped 1 previous similar message [ 2754.025480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.031021] Lustre: Skipped 2 previous similar messages [ 2759.137299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2759.140088] Lustre: Skipped 1 previous similar message [ 2764.257422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2764.263365] Lustre: Skipped 1 previous similar message [ 2768.351233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2768.584301] Lustre: server umount lustre-MDT0000 complete [ 2769.378297] LustreError: 81686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.394656] LustreError: 81686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2772.678804] LustreError: 81665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846776 with bad export cookie 2436744378690675991 [ 2772.682542] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.701427] LustreError: 81665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2773.222763] Lustre: server umount lustre-MDT0001 complete [ 2785.328383] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2788.859868] Key type lgssc unregistered [ 2789.182866] LNet: 85245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2789.201874] LNetError: 85245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2789.220277] LNet: Removed LNI 192.168.202.153@tcp [ 2790.212720] Key type .llcrypt unregistered [ 2790.217266] Key type ._llcrypt unregistered [ 2813.947167] Key type ._llcrypt registered [ 2813.949234] Key type .llcrypt registered [ 2814.104518] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2815.724790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2815.781898] alg: No test for adler32 (adler32-zlib) [ 2817.032412] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2817.363872] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2819.111438] Key type lgssc registered [ 2820.846570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2832.194056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2832.222806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.995601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2839.404536] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2844.128103] LustreError: 86653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.159652] LustreError: 86653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2849.061272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.261488] LustreError: 86654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2849.698222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2855.641901] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2864.939980] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2871.062837] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2878.186561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2878.545071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2879.602388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2887.219694] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2897.340969] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2905.040049] Lustre: server umount lustre-OST0000 complete [ 2909.664589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.665342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2909.678922] Lustre: Skipped 1 previous similar message [ 2909.703308] Lustre: Skipped 1 previous similar message [ 2914.789238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2914.792768] Lustre: Skipped 1 previous similar message [ 2919.003147] Lustre: server umount lustre-MDT0000 complete [ 2919.908452] LustreError: 86654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.925280] LustreError: 86654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2924.250585] LustreError: 86635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846927 with bad export cookie 2442559534287054851 [ 2924.256979] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2924.277111] LustreError: 86635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2933.856680] Lustre: server umount lustre-MDT0001 complete [ 2947.040407] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2950.248579] Key type lgssc unregistered [ 2950.607807] LNet: 90187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2950.616698] LNetError: 90187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2950.639622] LNet: Removed LNI 192.168.202.153@tcp [ 2951.703615] Key type .llcrypt unregistered [ 2951.708818] Key type ._llcrypt unregistered [ 2978.691377] Key type ._llcrypt registered [ 2978.696653] Key type .llcrypt registered [ 2978.872146] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2980.135306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2980.242297] alg: No test for adler32 (adler32-zlib) [ 2981.523882] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 2981.820718] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2983.527173] Key type lgssc registered [ 2985.104590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2993.053183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2993.076642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2994.553926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3000.279946] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3003.877066] LustreError: 91596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.901192] LustreError: 91596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3009.000578] LustreError: 91597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.464857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.756949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3015.243725] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3023.338545] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3030.636603] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3038.019232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.479449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.545958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3045.763505] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3054.110965] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3065.315772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3065.318373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.333188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3065.343114] Lustre: Skipped 1 previous similar message [ 3065.355305] Lustre: Skipped 1 previous similar message [ 3070.434612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3070.439803] Lustre: Skipped 1 previous similar message [ 3071.351222] Lustre: server umount lustre-OST0000 complete [ 3076.577792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3076.604143] LustreError: Skipped 1 previous similar message [ 3076.612448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.631970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.675171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.682699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.718711] Lustre: Skipped 1 previous similar message [ 3082.199587] Lustre: server umount lustre-MDT0000 complete [ 3085.793333] LustreError: 92326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.821845] LustreError: 92326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3086.338830] LustreError: 91578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847089 with bad export cookie 4378544318950082797 [ 3086.349595] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3086.670967] Lustre: server umount lustre-MDT0001 complete [ 3099.545290] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3104.161721] Key type lgssc unregistered [ 3104.592820] LNet: 95131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3104.605055] LNetError: 95131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3104.621624] LNet: Removed LNI 192.168.202.153@tcp [ 3105.806453] Key type .llcrypt unregistered [ 3105.809213] Key type ._llcrypt unregistered [ 3132.497473] Key type ._llcrypt registered [ 3132.500276] Key type .llcrypt registered [ 3132.606916] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3133.624833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3133.838763] alg: No test for adler32 (adler32-zlib) [ 3134.942799] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 3135.214488] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3136.880364] Key type lgssc registered [ 3138.306898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3146.183671] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3146.216108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3161.823438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3166.943411] LustreError: 96516:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 3193.632043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3200.655263] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3208.560137] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3214.424366] Lustre: server umount lustre-OST0000 complete [ 3223.006350] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:00:25 (1788847225) [ 3229.137100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3229.507821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3234.790214] LustreError: 98154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.820894] LustreError: 98154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3234.885819] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3238.883370] LustreError: 98155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3243.644540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.913542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3248.950669] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3256.079878] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3262.700946] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3269.959582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3270.383072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3274.509535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3276.729702] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3284.278206] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3299.296699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3299.302367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.319910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3299.354581] Lustre: Skipped 1 previous similar message [ 3304.422601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3304.431507] Lustre: Skipped 2 previous similar messages [ 3304.619530] Lustre: server umount lustre-OST0000 complete [ 3309.539177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.550651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.570133] Lustre: Skipped 1 previous similar message [ 3314.663299] LustreError: 98150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.665650] Lustre: server umount lustre-MDT0000 complete [ 3314.679732] LustreError: 98150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3318.572628] LustreError: 98136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847322 with bad export cookie 469810928105414986 [ 3318.581633] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3318.583996] LustreError: 98136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3318.904531] Lustre: server umount lustre-MDT0001 complete [ 3331.043686] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3334.698659] Key type lgssc unregistered [ 3335.047360] LNet: 101861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3335.062605] LNetError: 101861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3335.087866] LNet: Removed LNI 192.168.202.153@tcp [ 3336.136395] Key type .llcrypt unregistered [ 3336.142408] Key type ._llcrypt unregistered [ 3377.433569] Key type ._llcrypt registered [ 3377.435824] Key type .llcrypt registered [ 3377.556430] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3378.671660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3378.849271] alg: No test for adler32 (adler32-zlib) [ 3379.931341] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 3380.167264] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3381.863292] Key type lgssc registered [ 3382.819588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3388.520215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3388.527694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3389.885544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3394.224634] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3400.166637] LustreError: 104036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3400.204549] LustreError: 104036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3402.962632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3403.428721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3408.532127] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3416.345462] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3422.227286] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3428.504292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3429.078201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3435.278424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3436.386833] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3445.826601] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3459.043814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.044459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3459.057773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3462.817153] Lustre: server umount lustre-OST0000 complete [ 3469.282418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3469.295898] Lustre: Skipped 1 previous similar message [ 3469.302742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3469.307937] Lustre: Skipped 2 previous similar messages [ 3472.546375] Lustre: server umount lustre-MDT0000 complete [ 3474.417977] LustreError: 104036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3474.460977] LustreError: 104036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3476.794485] LustreError: 104018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847480 with bad export cookie 8283604376117266005 [ 3476.798899] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3476.806043] LustreError: 104018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3477.242165] Lustre: server umount lustre-MDT0001 complete [ 3488.448224] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3491.413699] Key type lgssc unregistered [ 3491.655586] LNet: 107568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3491.677770] LNetError: 107568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3491.697739] LNet: Removed LNI 192.168.202.153@tcp [ 3492.535098] Key type .llcrypt unregistered [ 3492.536960] Key type ._llcrypt unregistered [ 3512.515849] Key type ._llcrypt registered [ 3512.517685] Key type .llcrypt registered [ 3512.605730] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3513.603317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3513.662119] alg: No test for adler32 (adler32-zlib) [ 3514.740386] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 3515.051757] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3516.815287] Key type lgssc registered [ 3517.864720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3526.301801] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3526.345024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3541.856154] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3546.975770] LustreError: 108825:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 3573.663744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.112270] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3589.314464] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3595.783377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.102490] LustreError: MGC192.168.202.153@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3596.124378] LustreError: 109849:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3596.132950] LustreError: 109849:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3596.143628] LustreError: 109849:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3596.249909] Lustre: server umount lustre-MDT0000 complete [ 3596.259885] LustreError: 109849:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3612.790152] Lustre: server umount lustre-OST0000 complete [ 3623.442963] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 3636.521289] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3678.181630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.476662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3678.547507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3678.690436] Lustre: lustre-MDT0000: new disk, initializing [ 3678.830639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3678.863249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3684.636345] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3695.608992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.710597] Lustre: 112642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3695.759191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3695.768168] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3695.772482] Lustre: Skipped 1 previous similar message [ 3695.836825] Lustre: lustre-MDT0001: new disk, initializing [ 3695.954888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3695.986632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3696.002893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3700.937262] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3709.751479] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3715.434719] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3721.542385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3721.826634] Lustre: lustre-OST0000: new disk, initializing [ 3721.846326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3721.853636] Lustre: 114610:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3721.943835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3728.613380] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3730.484303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3730.496539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3730.654268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3737.600674] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3742.684179] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3742.904686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3748.656596] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3748.882446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3751.393243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3751.396989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3751.425808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3751.432894] Lustre: Skipped 1 previous similar message [ 3756.512751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3756.525487] Lustre: Skipped 1 previous similar message [ 3761.648866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3761.661753] Lustre: Skipped 1 previous similar message [ 3765.215157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3765.537336] Lustre: server umount lustre-OST0000 complete [ 3771.875435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.883852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3771.884920] Lustre: Skipped 2 previous similar messages [ 3782.113447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3782.131085] Lustre: Skipped 4 previous similar messages [ 3783.135161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3783.317379] Lustre: server umount lustre-MDT0000 complete [ 3787.440486] LustreError: 112634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847790 with bad export cookie 9750624163483603367 [ 3787.454587] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3787.743580] Lustre: server umount lustre-MDT0001 complete [ 3798.893689] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:10:00 (1788847800) [ 3804.156996] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3824.909941] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 3835.195505] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3870.734697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3870.919911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3870.943856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3871.010661] Lustre: lustre-MDT0000: new disk, initializing [ 3871.089251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3871.103899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3875.688817] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3886.175407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3886.332347] Lustre: 119516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3886.406096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3886.417382] Lustre: Skipped 1 previous similar message [ 3886.533989] Lustre: lustre-MDT0001: new disk, initializing [ 3886.662444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3886.693225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3886.722109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3891.505495] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3900.961757] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3906.719902] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3912.707505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3912.931426] Lustre: lustre-OST0000: new disk, initializing [ 3912.937153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3912.943674] Lustre: 121483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3913.009830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3919.262243] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3920.456963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3920.478243] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3920.648772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3928.356573] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3933.889403] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3934.137676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3939.235417] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3939.573742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3946.476870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3946.484286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.485617] LustreError: Skipped 1 previous similar message [ 3946.486849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3946.503603] Lustre: Skipped 1 previous similar message [ 3947.770484] Lustre: server umount lustre-OST0000 complete [ 3953.121793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3953.132424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.966820] Lustre: server umount lustre-MDT0000 complete [ 3961.363169] LustreError: 119509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847964 with bad export cookie 9750624163483604249 [ 3961.366175] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3961.373207] LustreError: 119509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3961.893405] Lustre: server umount lustre-MDT0001 complete [ 3970.490317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3971.181639] LustreError: 123700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3971.211682] LustreError: 123700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3971.333041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3975.795664] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3976.687040] LustreError: 123701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.772715] LustreError: 123700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3984.605064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3989.886872] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3997.004338] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4003.804594] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4011.998531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4019.239931] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4026.388699] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4035.746196] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4040.161263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4040.176747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.188760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4040.188767] Lustre: Skipped 4 previous similar messages [ 4040.199574] Lustre: Skipped 2 previous similar messages [ 4046.119402] Lustre: server umount lustre-OST0000 complete [ 4050.405464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.415551] Lustre: Skipped 1 previous similar message [ 4056.425395] Lustre: server umount lustre-MDT0000 complete [ 4060.332834] LustreError: 123682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848063 with bad export cookie 9750624163483605131 [ 4060.344152] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4060.747985] Lustre: server umount lustre-MDT0001 complete [ 4071.561704] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 4074.207857] Key type lgssc unregistered [ 4074.528906] LNet: 127328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4074.534855] LNetError: 127328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4074.558318] LNet: Removed LNI 192.168.202.153@tcp [ 4075.515513] Key type .llcrypt unregistered [ 4075.517436] Key type ._llcrypt unregistered [ 4095.746983] Key type ._llcrypt registered [ 4095.754858] Key type .llcrypt registered [ 4096.072573] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 4115.378488] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 4116.446380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4116.693111] alg: No test for adler32 (adler32-zlib) [ 4117.760438] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 4117.954737] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 4119.655236] Key type lgssc registered [ 4120.695524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4156.579859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4156.612432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4157.940666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4157.992812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4158.143715] Lustre: lustre-MDT0000: new disk, initializing [ 4158.255331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.281393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4163.691497] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4175.972755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.090728] Lustre: 130478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4176.131397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4176.134070] Lustre: Skipped 1 previous similar message [ 4176.186235] Lustre: lustre-MDT0001: new disk, initializing [ 4176.250058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4176.271575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4176.275265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4181.652191] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4191.487655] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4197.609280] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4204.930228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4205.250672] Lustre: lustre-OST0000: new disk, initializing [ 4205.255288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4205.260757] Lustre: 132447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4205.337549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.790798] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4213.815593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4213.822155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4213.918339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4220.606746] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4225.022873] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4225.135562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4229.774932] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4230.120463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4234.722037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4234.726088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.741677] LustreError: Skipped 1 previous similar message [ 4234.744164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4234.765174] Lustre: Skipped 1 previous similar message [ 4239.038115] Lustre: server umount lustre-OST0000 complete [ 4244.967696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4244.998884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4245.001819] Lustre: Skipped 1 previous similar message [ 4249.515856] Lustre: server umount lustre-MDT0000 complete [ 4253.209043] LustreError: 130470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848256 with bad export cookie 3722552974852161851 [ 4253.217926] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4253.222257] LustreError: 130470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4253.735742] Lustre: server umount lustre-MDT0001 complete [ 4262.480500] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:17:44 (1788848264) [ 4268.919101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.502214] LustreError: 134858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.515343] LustreError: 134858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4269.561336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4274.325703] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4274.663744] LustreError: 134859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4278.820623] LustreError: 134858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4282.633282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4282.980810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4287.858958] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4295.423607] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4301.102486] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4303.053335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.076642] Lustre: Skipped 1 previous similar message [ 4303.095230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4303.097811] Lustre: Skipped 1 previous similar message [ 4308.257518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4308.267016] Lustre: Skipped 1 previous similar message [ 4309.433434] Lustre: server umount lustre-MDT0000 complete [ 4313.335869] LustreError: 134838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848316 with bad export cookie 3722552974852162733 [ 4313.346544] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4313.362148] LustreError: 134838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4313.615970] Lustre: server umount lustre-MDT0001 complete [ 4321.742624] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:18:44 (1788848324) [ 4329.300373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4344.927692] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4350.047420] LustreError: 137273:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 4375.775497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.050263] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4389.880667] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4391.836147] Lustre: server umount lustre-OST0000 complete [ 4400.196586] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:20:02 (1788848402) [ 4406.690977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.207517] LustreError: 138727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.280371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.763591] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4412.388555] LustreError: 138728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4419.595574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4425.288539] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4433.604316] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4439.266354] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4446.751718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4447.161636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4447.169919] Lustre: Skipped 1 previous similar message [ 4453.288474] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4460.969777] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4474.856021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.859147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4474.864679] Lustre: Skipped 3 previous similar messages [ 4474.873760] Lustre: Skipped 2 previous similar messages [ 4475.170084] Lustre: server umount lustre-MDT0000 complete [ 4478.816305] LustreError: 140571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848482 with bad export cookie 3722552974852163398 [ 4478.820923] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.827243] LustreError: 140571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4479.100762] Lustre: server umount lustre-MDT0001 complete [ 4493.683837] Lustre: server umount lustre-OST0000 complete [ 4503.558836] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:21:45 (1788848505) [ 4510.528529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.097401] LustreError: 142400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4511.122769] LustreError: 142400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4511.218906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4516.297987] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4525.447470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4528.990397] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4535.683795] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4544.652564] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4554.100943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4556.446831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4562.599900] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4570.418695] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4576.074968] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4576.512909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4581.471827] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4581.783203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4586.466714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4586.473674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.474645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4586.504713] Lustre: Skipped 1 previous similar message [ 4598.239294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4598.456042] Lustre: server umount lustre-OST0000 complete [ 4606.944265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4606.967809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4606.970432] Lustre: Skipped 5 previous similar messages [ 4608.840942] Lustre: server umount lustre-MDT0000 complete [ 4612.064285] LustreError: 143131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4612.095390] LustreError: 143131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4612.747943] LustreError: 142382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848616 with bad export cookie 3722552974852164462 [ 4612.748561] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4612.755479] LustreError: 142382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4613.220840] Lustre: server umount lustre-MDT0001 complete [ 4622.173667] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:23:44 (1788848624) [ 4629.338476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4644.895996] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4650.015922] LustreError: 146563:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 4675.743299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4675.757421] Lustre: Skipped 2 previous similar messages [ 4683.128990] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4691.047951] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4698.124597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.670351] LustreError: 146588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4698.924143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4703.899563] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4712.647335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4717.652518] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4725.486100] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4731.374984] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4736.152752] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4736.417904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4741.309224] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4741.534260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4743.658220] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.672173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4743.689060] Lustre: Skipped 1 previous similar message [ 4743.694057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4749.668379] Lustre: server umount lustre-MDT0000 complete [ 4753.395485] LustreError: 146569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848756 with bad export cookie 3722552974852165491 [ 4753.401790] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4753.411957] LustreError: 146569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4753.809057] Lustre: server umount lustre-MDT0001 complete [ 4768.466761] Lustre: server umount lustre-OST0000 complete [ 4777.680847] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:26:19 (1788848779) [ 4785.007549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4800.609747] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4805.729059] LustreError: 150756:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 4837.559123] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4845.985771] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4853.483025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4853.920758] LustreError: 150781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.955604] LustreError: 150781:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4854.147239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4859.485146] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4868.572917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4874.114187] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4881.912963] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4887.503447] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4894.204757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4894.396495] Lustre: lustre-OST0001: new disk, initializing [ 4894.400781] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4894.406306] Lustre: 153630:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4899.944354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4899.957067] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4900.152475] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4900.339028] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4907.706190] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4911.953411] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4912.163485] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4917.146643] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4917.397730] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4920.290086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4920.297997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.324722] Lustre: Skipped 2 previous similar messages [ 4920.340072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4920.349414] Lustre: Skipped 5 previous similar messages [ 4925.376529] Lustre: server umount lustre-OST0000 complete [ 4931.053503] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4931.062093] LustreError: Skipped 1 previous similar message [ 4934.811959] Lustre: server umount lustre-OST0001 complete [ 4940.769636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4940.780683] LustreError: Skipped 1 previous similar message [ 4944.576337] Lustre: server umount lustre-MDT0000 complete [ 4948.571435] LustreError: 153629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848952 with bad export cookie 3722552974852166443 [ 4948.578953] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4948.582691] LustreError: 153629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4949.144325] Lustre: server umount lustre-MDT0001 complete [ 4970.830863] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:29:33 (1788848973) [ 4972.374443] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4974.315045] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4975.952735] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:29:38 (1788848978) [ 4982.243041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4982.486536] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4982.519348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4982.682833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4982.690959] Lustre: Skipped 6 previous similar messages [ 4986.982266] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 4993.907859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4994.047801] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4994.091996] Lustre: 157190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4998.036040] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5004.545585] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5008.969823] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5014.178404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5014.294053] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5020.088680] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5021.332469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5027.415851] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5032.162235] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5032.339171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5036.674598] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5036.845293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5041.634088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5041.646098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5041.659456] Lustre: Skipped 5 previous similar messages [ 5045.031262] Lustre: server umount lustre-OST0000 complete [ 5065.296374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5066.728384] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.740565] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5071.160349] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5077.931535] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5082.086804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.097622] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5082.503831] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5082.729667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5086.571493] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5086.738468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5091.323860] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a334794f000.ost_server_uuid 50 [ 5092.795470] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9a334794f000.ost_server_uuid in FULL state after 0 sec [ 5092.968053] LustreError: 158454:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.53@tcp) returned error from blocking AST (req@000000004e1f1a84 x1875743207583232 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b1add98e/0x33a92a352ba31c85 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.53@tcp remote: 0x8934829eb494fe1b expref: 5 pid: 157197 timeout: 5192 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5092.997585] LustreError: lustre-MDT0000: A client on nid 192.168.202.53@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5093.003875] LustreError: 157185:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.202.53@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b1add98e/0x33a92a352ba31c85 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.53@tcp remote: 0x8934829eb494fe1b expref: 6 pid: 157197 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5097.443491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5097.443560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5097.451339] LustreError: Skipped 1 previous similar message [ 5097.474144] Lustre: Skipped 10 previous similar messages [ 5103.317343] Lustre: server umount lustre-OST0000 complete [ 5113.455719] Lustre: server umount lustre-MDT0000 complete [ 5117.009325] LustreError: 159051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849120 with bad export cookie 3722552974852167619 [ 5117.017316] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5117.020812] LustreError: 159051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5123.528633] Lustre: server umount lustre-MDT0001 complete [ 5134.744988] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 5137.257788] Key type lgssc unregistered [ 5137.525541] LNet: 163302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5137.529932] LNetError: 163302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5137.535614] LNet: Removed LNI 192.168.202.153@tcp [ 5138.062949] Key type .llcrypt unregistered [ 5138.066262] Key type ._llcrypt unregistered [ 5162.825480] Key type ._llcrypt registered [ 5162.827550] Key type .llcrypt registered [ 5163.021807] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5164.183293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5164.230917] alg: No test for adler32 (adler32-zlib) [ 5165.307416] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 5165.659343] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 5167.367179] Key type lgssc registered [ 5168.832956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5175.923887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5175.945263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.364830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5181.836357] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5187.554187] LustreError: 164716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5187.585279] LustreError: 164716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5190.255224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.621288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5194.595529] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5201.842548] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5207.555083] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5214.317359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5214.637664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5220.244779] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5221.827393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5227.840918] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5233.082855] Lustre: Failing over lustre-MDT0000 [ 5233.319915] Lustre: server umount lustre-MDT0000 complete [ 5234.569124] LustreError: 164712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5236.706179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5236.717266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.726302] LustreError: 164717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.223189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.250344] Lustre: Skipped 1 previous similar message [ 5242.338057] LustreError: 166556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5242.367561] LustreError: 166556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5245.402320] LustreError: 164697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.53@tcp arrived at 1788849248 with bad export cookie 11281694135220430090 [ 5245.417269] LustreError: 164697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5247.462077] LustreError: 164711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5247.494522] LustreError: 164711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5251.896185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.073468] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5252.418395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5252.468416] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5256.845606] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5257.701464] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5257.709513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5259.086323] Lustre: lustre-MDT0000: Denying connection for new client e3ba140a-67b4-4cb5-aa14-281780fb1763 (at 192.168.202.53@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5266.912754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5266.914948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.919966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5267.947881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5273.059307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5273.067692] Lustre: Skipped 2 previous similar messages [ 5278.187067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5278.200941] Lustre: Skipped 1 previous similar message [ 5281.247302] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5281.409712] Lustre: server umount lustre-OST0000 complete [ 5284.607523] LustreError: 168444:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5284.623769] Lustre: 167792:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5284.635414] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5284.657235] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5284.685089] LustreError: 167792:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000909733fc x1875744306278784/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5284.709917] LustreError: 167792:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5284.729887] LustreError: 167792:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5284.740820] Lustre: lustre-MDT0000: Recovery over after 0:27, of 2 clients 0 recovered and 2 were evicted. [ 5285.078808] Lustre: server umount lustre-MDT0000 complete [ 5288.416084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5288.432486] Lustre: Skipped 1 previous similar message [ 5288.435091] LustreError: 164713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.876258] LustreError: 166560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849292 with bad export cookie 11281694135220430188 [ 5288.884446] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5288.894787] LustreError: 166560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5291.328615] Lustre: server umount lustre-MDT0001 complete [ 5301.654927] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 5304.695915] Key type lgssc unregistered [ 5304.998410] LNet: 169194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5305.006391] LNetError: 169194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5306.032794] LNet: Removed LNI 192.168.202.153@tcp [ 5306.769864] Key type .llcrypt unregistered [ 5306.772048] Key type ._llcrypt unregistered [ 5328.880654] Key type ._llcrypt registered [ 5328.882376] Key type .llcrypt registered [ 5328.988413] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5329.869563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5330.018040] alg: No test for adler32 (adler32-zlib) [ 5331.027475] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 5331.243512] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 5332.978069] Key type lgssc registered [ 5334.029783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5340.439170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5340.469289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5341.819594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5345.078400] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5351.477380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5351.697029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5354.939502] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5359.991843] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5364.525719] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5369.586974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5369.885234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5375.245516] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5380.944124] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5388.348719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5397.483948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5397.490271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5397.498716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5398.501997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5398.510795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5398.511042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.526983] Lustre: Skipped 1 previous similar message [ 5402.557881] Lustre: server umount lustre-OST0000 complete [ 5408.224231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5408.229113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5408.243565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5411.687516] Lustre: server umount lustre-MDT0000 complete [ 5413.856853] LustreError: 170603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.891258] LustreError: 170603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5415.289473] LustreError: 170584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849418 with bad export cookie 16739833237741937343 [ 5415.298762] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5415.307579] LustreError: 170584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5415.905770] Lustre: server umount lustre-MDT0001 complete [ 5425.479632] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 5428.202337] Key type lgssc unregistered [ 5428.468629] LNet: 174152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5428.487961] LNetError: 174152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5429.542792] LNet: Removed LNI 192.168.202.153@tcp [ 5430.208547] Key type .llcrypt unregistered [ 5430.213543] Key type ._llcrypt unregistered [ 5462.539612] Key type ._llcrypt registered [ 5462.542368] Key type .llcrypt registered [ 5462.659335] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5463.611302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5463.720434] alg: No test for adler32 (adler32-zlib) [ 5464.879765] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 5465.108161] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 5466.760466] Key type lgssc registered [ 5468.022741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5475.141613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5475.161064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5476.607650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5481.010458] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5487.082833] LustreError: 175854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5487.109349] LustreError: 175854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5488.984975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.251922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5493.912359] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5500.322458] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5505.084869] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5511.210144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5511.513037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5516.743514] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5521.772932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5522.473725] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5529.056806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5529.063087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.081765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5532.130278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5532.136770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5532.143125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5532.166091] Lustre: Skipped 1 previous similar message [ 5537.254911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5537.258740] Lustre: Skipped 1 previous similar message [ 5541.343264] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5541.468901] Lustre: server umount lustre-OST0000 complete [ 5545.441524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5545.448122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.455265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5547.489711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.685663] Lustre: server umount lustre-MDT0000 complete [ 5552.616164] LustreError: 175850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5552.633688] LustreError: 175850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5553.754661] LustreError: 175834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849557 with bad export cookie 17398118181962123618 [ 5553.763254] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5553.768417] LustreError: 175834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5554.152766] Lustre: server umount lustre-MDT0001 complete [ 5562.072651] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 5564.322388] Key type lgssc unregistered [ 5564.574293] LNet: 179383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5564.578557] LNetError: 179383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5565.609927] LNet: Removed LNI 192.168.202.153@tcp [ 5566.461782] Key type .llcrypt unregistered [ 5566.463819] Key type ._llcrypt unregistered [ 5584.374583] Key type ._llcrypt registered [ 5584.376656] Key type .llcrypt registered [ 5584.463500] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5585.482595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5585.522821] alg: No test for adler32 (adler32-zlib) [ 5586.562038] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 5586.800183] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 5588.455418] Key type lgssc registered [ 5589.491604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5599.235750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5599.259700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5600.757895] Lustre: *** cfs_fail_loc=135, val=0*** [ 5600.873872] LustreError: 180883:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5600.884256] LustreError: 180883:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.153@tcp: cfg command failed: rc = -2 [ 5600.892157] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5600.892157] [ 5600.902735] LustreError: MGC192.168.202.153@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5600.918352] LustreError: 180845:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5600.928229] LustreError: 180845:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5600.930821] LustreError: 180845:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5600.939302] LustreError: 180845:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5601.038797] Lustre: server umount lustre-MDT0000 complete [ 5601.042895] LustreError: 180845:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5606.537227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.836799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5611.664779] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5617.123209] LustreError: 181254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5617.154222] LustreError: 181254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5619.778479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5620.061548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5624.622955] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5631.479853] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5636.123329] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5640.675141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5640.679108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5640.686134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.720327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.726698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5644.062742] Lustre: server umount lustre-MDT0000 complete [ 5644.769942] LustreError: 181254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5644.783416] LustreError: 181254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5647.154704] LustreError: 181235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849650 with bad export cookie 5392041870076823756 [ 5647.165452] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5647.166795] LustreError: 181235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5647.446743] Lustre: server umount lustre-MDT0001 complete [ 5655.255663] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 5657.388533] Key type lgssc unregistered [ 5657.727146] LNet: 183573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5657.732335] LNetError: 183573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5657.749752] LNet: Removed LNI 192.168.202.153@tcp [ 5658.508111] Key type .llcrypt unregistered [ 5658.509691] Key type ._llcrypt unregistered [ 5675.027069] Key type ._llcrypt registered [ 5675.029777] Key type .llcrypt registered [ 5675.130632] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 5675.371426] Key type .llcrypt unregistered [ 5675.373537] Key type ._llcrypt unregistered [ 5688.994533] Key type ._llcrypt registered [ 5688.996267] Key type .llcrypt registered [ 5689.054779] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5689.899456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5689.909374] alg: No test for adler32 (adler32-zlib) [ 5690.920288] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 5691.095988] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 5692.711261] Key type lgssc registered [ 5693.712803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5700.304405] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5700.332757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5715.871561] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5720.991985] LustreError: 185677:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 5747.679240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5752.821917] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5758.777180] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5764.585513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5764.970447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5765.015202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5768.749755] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5770.212395] LustreError: 185702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.232033] LustreError: 185702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5775.329854] LustreError: 185703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.071706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5776.355802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5780.435252] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5786.439747] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5790.526739] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5792.827586] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5806.049358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5806.054962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.076750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5807.072162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5807.080531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5807.088390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.097942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.219619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.226391] Lustre: Skipped 1 previous similar message [ 5818.335163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5818.341117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5818.358449] Lustre: Skipped 1 previous similar message [ 5818.484928] Lustre: server umount lustre-OST0000 complete [ 5822.432205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5822.437669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.821419] Lustre: server umount lustre-MDT0000 complete [ 5828.583328] LustreError: 186715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.598210] LustreError: 186715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5830.528116] LustreError: 185683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788849833 with bad export cookie 12253062475350011391 [ 5830.540368] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5830.751591] Lustre: server umount lustre-MDT0001 complete [ 5840.018412] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 5842.451476] Key type lgssc unregistered [ 5842.758631] LNet: 189689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5842.769135] LNetError: 189689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5842.780766] LNet: Removed LNI 192.168.202.153@tcp [ 5843.482047] Key type .llcrypt unregistered [ 5843.491154] Key type ._llcrypt unregistered [ 5864.550097] Key type ._llcrypt registered [ 5864.556977] Key type .llcrypt registered [ 5864.655275] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 5865.982643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5865.993587] alg: No test for adler32 (adler32-zlib) [ 5867.078623] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 5867.282559] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 5868.959161] Key type lgssc registered [ 5870.067145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5877.737894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5877.758996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.261339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5883.818184] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5889.508140] LustreError: 191075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.518171] LustreError: 191075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5890.824039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5891.231857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5895.099275] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5900.607695] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5904.819499] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5911.462077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5911.825319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5916.825255] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5922.483530] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5924.157137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5931.815039] Lustre: Failing over lustre-MDT0000 [ 5932.017885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5932.021679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5932.039772] LustreError: 191075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.076886] Lustre: server umount lustre-MDT0000 complete [ 5934.486332] LustreError: 191070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.568723] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5939.580852] LustreError: 191070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5939.593207] LustreError: 191070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5944.715764] LustreError: 192919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5944.739404] LustreError: 192919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5949.925747] Lustre: 190388:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788849937/real 1788849937] req@00000000c0e08264 x1875745041309952/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788849953 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5949.952947] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5951.945382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5954.874180] LustreError: 193821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.891217] LustreError: 193821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5960.162422] LustreError: 190386:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000099111314 x1875745041315328/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5960.420436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5960.474405] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5961.039439] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5964.676487] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 5965.814972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5965.856659] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5965.901707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5968.755784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5977.443689] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5982.688647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5982.709244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.725914] Lustre: Skipped 1 previous similar message [ 5982.731702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5986.274549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5986.284713] Lustre: Skipped 2 previous similar messages [ 5988.311655] Lustre: server umount lustre-OST0000 complete [ 5996.517570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5996.522635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.523619] LustreError: Skipped 1 previous similar message [ 5996.524610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5996.533016] Lustre: Skipped 2 previous similar messages [ 6001.633124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6001.639727] Lustre: Skipped 2 previous similar messages [ 6006.239394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6006.757151] LustreError: 191076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6006.769832] LustreError: 191076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6006.866785] Lustre: server umount lustre-MDT0000 complete [ 6010.442853] LustreError: 191057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850013 with bad export cookie 3930056424169785185 [ 6010.444733] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6010.448728] LustreError: 191057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6010.719647] Lustre: server umount lustre-MDT0001 complete [ 6021.877405] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 6024.387641] Key type lgssc unregistered [ 6024.618142] LNet: 196012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6024.626960] LNetError: 196012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6024.649072] LNet: Removed LNI 192.168.202.153@tcp [ 6025.347727] Key type .llcrypt unregistered [ 6025.350314] Key type ._llcrypt unregistered [ 6044.304443] Key type ._llcrypt registered [ 6044.307204] Key type .llcrypt registered [ 6044.431397] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 6045.527876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6045.554363] alg: No test for adler32 (adler32-zlib) [ 6046.555218] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 6046.745975] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 6048.343241] Key type lgssc registered [ 6049.443575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6056.584888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6056.609768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.969425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.399992] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6068.192560] LustreError: 197399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6068.200347] LustreError: 197399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6069.562679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.957619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6074.527728] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6080.721622] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6085.804927] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6091.136969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6091.457790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6097.112806] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6102.929684] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6103.818911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6106.379749] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6112.555554] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6122.862830] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6129.643911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6129.659750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6129.670800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6129.678523] Lustre: Skipped 1 previous similar message [ 6129.692802] Lustre: Skipped 1 previous similar message [ 6134.637636] Lustre: server umount lustre-OST0000 complete [ 6139.873054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6139.878952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6139.880427] Lustre: Skipped 1 previous similar message [ 6139.890351] Lustre: Skipped 1 previous similar message [ 6144.200591] Lustre: server umount lustre-MDT0000 complete [ 6144.993836] LustreError: 198652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.014504] LustreError: 198652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6147.296121] LustreError: 200188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850150 with bad export cookie 2551331278865243592 [ 6147.296517] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6147.305315] LustreError: 200188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6147.669424] Lustre: server umount lustre-MDT0001 complete [ 6156.700699] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 6158.944604] Key type lgssc unregistered [ 6159.218914] LNet: 201243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6159.229621] LNetError: 201243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6159.247323] LNet: Removed LNI 192.168.202.153@tcp [ 6159.963340] Key type .llcrypt unregistered [ 6159.964976] Key type ._llcrypt unregistered [ 6175.214866] Key type ._llcrypt registered [ 6175.218471] Key type .llcrypt registered [ 6175.327163] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 6176.220825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6176.302647] alg: No test for adler32 (adler32-zlib) [ 6177.300217] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 6177.523149] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 6179.184891] Key type lgssc registered [ 6180.151293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6186.668492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6186.692118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.978081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6191.140927] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6198.157349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.240060] LustreError: 202484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6198.265341] LustreError: 202484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6198.504803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6202.162260] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6207.999186] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6212.253421] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6217.416844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6217.725865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6222.490656] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6227.322551] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6233.072173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6236.616239] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6245.805857] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6255.831544] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6265.524620] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6274.018076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6274.022992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6274.027855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6274.051111] Lustre: Skipped 1 previous similar message [ 6279.143327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.147955] Lustre: Skipped 2 previous similar messages [ 6279.311289] Lustre: server umount lustre-OST0000 complete [ 6284.259808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.263447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6284.281710] Lustre: Skipped 1 previous similar message [ 6284.294185] Lustre: Skipped 1 previous similar message [ 6288.480429] Lustre: server umount lustre-MDT0000 complete [ 6289.376334] LustreError: 202484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6289.402259] LustreError: 202484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6291.724686] LustreError: 202463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850295 with bad export cookie 5933464773121927906 [ 6291.747295] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6291.750582] LustreError: 202463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6291.997511] Lustre: server umount lustre-MDT0001 complete [ 6300.483699] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 6302.910724] Key type lgssc unregistered [ 6303.210081] LNet: 207597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6303.214546] LNetError: 207597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6303.247535] LNet: Removed LNI 192.168.202.153@tcp [ 6304.041719] Key type .llcrypt unregistered [ 6304.043755] Key type ._llcrypt unregistered [ 6321.477443] Key type ._llcrypt registered [ 6321.478949] Key type .llcrypt registered [ 6321.575317] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 6322.411946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6322.433131] alg: No test for adler32 (adler32-zlib) [ 6323.514599] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 6323.715405] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 6325.351741] Key type lgssc registered [ 6326.189526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6331.972977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6331.983345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.224514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6336.452851] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6342.627358] LustreError: 208986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6342.648524] LustreError: 208986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6343.162524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6343.485106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6347.046243] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6352.199424] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6356.294833] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6361.467500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6361.817175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.567466] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6371.922732] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6378.232370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6393.824215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6393.824552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.831083] LustreError: Skipped 1 previous similar message [ 6393.832137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6393.867094] Lustre: Skipped 1 previous similar message [ 6397.344426] Lustre: server umount lustre-OST0000 complete [ 6399.972408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6399.979058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6399.991863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6399.994629] Lustre: Skipped 1 previous similar message [ 6404.067031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6404.067140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6409.185108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6409.190108] Lustre: Skipped 2 previous similar messages [ 6414.304638] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6414.305195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6414.316395] Lustre: Skipped 1 previous similar message [ 6414.477459] Lustre: server umount lustre-MDT0000 complete [ 6417.109280] LustreError: 208967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850420 with bad export cookie 10319697324316077930 [ 6417.113564] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6417.119736] LustreError: 208967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6417.340656] Lustre: server umount lustre-MDT0001 complete [ 6424.302369] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 6426.658192] Key type lgssc unregistered [ 6426.930795] LNet: 212686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6426.936545] LNetError: 212686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6426.950178] LNet: Removed LNI 192.168.202.153@tcp [ 6427.616336] Key type .llcrypt unregistered [ 6427.618471] Key type ._llcrypt unregistered [ 6440.719259] Key type ._llcrypt registered [ 6440.721351] Key type .llcrypt registered [ 6440.787615] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 6441.485336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6441.577201] alg: No test for adler32 (adler32-zlib) [ 6442.550540] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 6442.732105] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 6444.359367] Key type lgssc registered [ 6445.188300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6450.178619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6450.191843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6451.445559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6453.682482] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6458.671610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6458.891979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6461.481846] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6466.181548] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6470.260645] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6474.874464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6475.179525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6476.209503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6478.979034] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6483.909348] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6494.690440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6494.692421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6494.692803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6494.710358] Lustre: Skipped 1 previous similar message [ 6499.818276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6499.822269] Lustre: Skipped 2 previous similar messages [ 6500.088929] Lustre: server umount lustre-OST0000 complete [ 6504.928438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6504.934167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6504.935873] Lustre: Skipped 1 previous similar message [ 6509.245100] Lustre: server umount lustre-MDT0000 complete [ 6510.048735] LustreError: 213929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6510.058791] LustreError: 213929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6511.815542] LustreError: 213910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850515 with bad export cookie 17443834733215080425 [ 6511.817144] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6511.821726] LustreError: 213910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6512.037709] Lustre: server umount lustre-MDT0001 complete [ 6519.116601] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 6520.998153] Key type lgssc unregistered [ 6521.252994] LNet: 217606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6521.258921] LNetError: 217606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6522.283320] LNet: Removed LNI 192.168.202.153@tcp [ 6522.891436] Key type .llcrypt unregistered [ 6522.893625] Key type ._llcrypt unregistered [ 6535.631442] Key type ._llcrypt registered [ 6535.632954] Key type .llcrypt registered [ 6535.700806] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 6536.340929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6536.385046] alg: No test for adler32 (adler32-zlib) [ 6537.396753] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 6537.557676] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 6539.192867] Key type lgssc registered [ 6539.941343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6545.375891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6545.390225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6546.707073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6549.697481] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6555.018277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6555.270153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6557.685831] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6561.407415] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6564.154475] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6568.117307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6568.365475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6571.897767] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6572.458173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6576.361572] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6582.920372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6582.969898] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6583.107952] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6586.346624] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6591.236363] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6592.353282] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6603.785516] Lustre: Permanently deactivating lustre-OST0001 [ 6603.800678] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6614.078289] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6618.039843] Lustre: Permanently reactivating lustre-OST0001 [ 6618.053722] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6618.056494] Lustre: Skipped 2 previous similar messages [ 6623.248253] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.258191] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6623.263696] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6623.271948] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6623.294401] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6632.186706] Lustre: server umount lustre-OST0001 complete [ 6633.441757] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6633.447749] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6633.463246] Lustre: Skipped 1 previous similar message [ 6633.472643] LustreError: 220667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6633.491576] LustreError: 220667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6634.985332] LustreError: 221086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6634.993332] LustreError: 221086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6636.001327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6636.012327] LustreError: Skipped 1 previous similar message [ 6636.014433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.026123] Lustre: Skipped 1 previous similar message [ 6636.032233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6639.075801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6639.087977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6639.110755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6640.101557] LustreError: 221086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6640.112486] LustreError: 221086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6641.605391] Lustre: server umount lustre-OST0000 complete [ 6645.217422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6645.219168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6645.228779] Lustre: Skipped 2 previous similar messages [ 6645.231239] Lustre: Skipped 1 previous similar message [ 6650.339252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6650.347633] Lustre: Skipped 2 previous similar messages [ 6650.454764] Lustre: server umount lustre-MDT0000 complete [ 6653.420376] LustreError: 222464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850656 with bad export cookie 3932891218663425398 [ 6653.423504] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6653.427697] LustreError: 222464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6653.648658] Lustre: server umount lustre-MDT0001 complete [ 6668.166756] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 6671.446702] Key type lgssc unregistered [ 6672.071766] LNet: 223909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6672.090477] LNetError: 223909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6672.119252] LNet: Removed LNI 192.168.202.153@tcp [ 6673.358296] Key type .llcrypt unregistered [ 6673.360518] Key type ._llcrypt unregistered [ 6726.392455] Key type ._llcrypt registered [ 6726.394234] Key type .llcrypt registered [ 6726.570684] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 6728.122597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6728.148794] alg: No test for adler32 (adler32-zlib) [ 6729.235285] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 6729.494848] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 6731.263742] Key type lgssc registered [ 6732.592949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6741.460957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6741.486618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6742.715431] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6742.767724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6742.992152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6747.478259] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6755.844786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.980947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6756.039667] Lustre: 226122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6756.288141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6760.858478] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6767.848776] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6772.812266] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6778.785736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6778.949624] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6779.199796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.017478] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 6787.171814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6792.478487] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6795.966909] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6800.536458] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6808.501108] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6817.333961] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6830.627291] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6830.633291] Lustre: Skipped 1 previous similar message [ 6849.621727] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6849.628681] Lustre: Skipped 1 previous similar message [ 6885.542977] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6885.546093] Lustre: Skipped 4 previous similar messages [ 6958.970720] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6958.975404] Lustre: Skipped 8 previous similar messages [ 6966.250498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6966.265388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6966.279399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6966.752352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6971.565525] Lustre: server umount lustre-OST0000 complete [ 6976.481354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6976.491767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.507108] Lustre: Skipped 1 previous similar message [ 6976.518599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6976.525782] Lustre: Skipped 1 previous similar message [ 6982.113978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6982.134830] Lustre: Skipped 2 previous similar messages [ 6987.235191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6987.272236] Lustre: Skipped 1 previous similar message [ 6990.303508] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6990.530650] Lustre: server umount lustre-MDT0000 complete [ 6993.969155] LustreError: 229135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850997 with bad export cookie 7475320149588073565 [ 6993.972156] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6993.981093] LustreError: 229135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6994.325328] Lustre: server umount lustre-MDT0001 complete [ 7004.963858] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7008.116271] Key type lgssc unregistered [ 7008.466609] LNet: 230923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7008.470459] LNetError: 230923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7008.499337] LNet: Removed LNI 192.168.202.153@tcp [ 7009.302519] Key type .llcrypt unregistered [ 7009.304409] Key type ._llcrypt unregistered [ 7033.582047] Key type ._llcrypt registered [ 7033.590747] Key type .llcrypt registered [ 7033.778404] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7034.851787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7034.925794] alg: No test for adler32 (adler32-zlib) [ 7036.041713] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 7036.406791] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 7038.159524] Key type lgssc registered [ 7039.602984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7047.637149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7047.667742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7049.249794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7054.351148] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7059.452475] LustreError: 232337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.491597] LustreError: 232337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7064.463903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7064.546171] LustreError: 232336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7064.960286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7069.715333] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7077.440494] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7083.648791] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7090.238744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7090.733134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7093.818414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7096.648458] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 7104.391745] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7123.467562] Lustre: server umount lustre-OST0000 complete [ 7130.600056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7130.607761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7130.625329] Lustre: Skipped 1 previous similar message [ 7135.720688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7135.726721] Lustre: Skipped 2 previous similar messages [ 7137.870920] Lustre: server umount lustre-MDT0000 complete [ 7140.844136] LustreError: 233731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.846098] Lustre: 231644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851128/real 1788851128] req@00000000a11935d2 x1875746267107584/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788851144 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7140.868333] LustreError: 233731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7140.922975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.869651] LustreError: 234985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851145 with bad export cookie 12648000898853373187 [ 7141.883466] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7141.883612] LustreError: 234985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7142.640371] Lustre: server umount lustre-MDT0001 complete [ 7154.596491] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7157.129922] Key type lgssc unregistered [ 7157.494173] LNet: 236023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7157.501866] LNetError: 236023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7157.532793] LNet: Removed LNI 192.168.202.153@tcp [ 7158.304420] Key type .llcrypt unregistered [ 7158.305801] Key type ._llcrypt unregistered [ 7183.698384] Key type ._llcrypt registered [ 7183.700760] Key type .llcrypt registered [ 7183.931774] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7186.159557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7186.199347] alg: No test for adler32 (adler32-zlib) [ 7187.321368] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 7187.694933] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 7189.351194] Key type lgssc registered [ 7190.914882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7462.646963] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7465.668975] Key type lgssc unregistered [ 7465.941382] LNet: 238042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7465.948744] LNetError: 238042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7465.958030] LNet: Removed LNI 192.168.202.153@tcp [ 7466.933510] Key type .llcrypt unregistered [ 7466.935148] Key type ._llcrypt unregistered [ 7494.573326] Key type ._llcrypt registered [ 7494.578127] Key type .llcrypt registered [ 7494.700306] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 7512.704781] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 7513.990663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7514.177128] alg: No test for adler32 (adler32-zlib) [ 7515.247311] Lustre: Lustre: Build Version: 2.17.57_111_g33f3974 [ 7515.610406] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 7517.423488] Key type lgssc registered [ 7519.267643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7553.365704] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7256 sec ========== 03:12:35 (1788851555) [ 7555.356964] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:12:37 (1788851557) === [ 7559.084107] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:12:41 (1788851561) === [ 7606.758951] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 7609.884862] Key type lgssc unregistered [ 7610.187345] LNet: 243274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7610.198769] LNetError: 243274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7610.229951] LNet: Removed LNI 192.168.202.153@tcp [ 7611.331445] Key type .llcrypt unregistered [ 7611.332759] Key type ._llcrypt unregistered