[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 622551456 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003217] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010138] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013065] *** VALIDATE selinux *** [ 0.021037] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026209] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027302] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029096] *** VALIDATE tmpfs *** [ 0.030380] *** VALIDATE proc *** [ 0.031238] *** VALIDATE cgroup *** [ 0.032016] *** VALIDATE cgroup2 *** [ 0.034115] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035188] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037043] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041436] debug: unmapping init [mem 0xffffffffb3259000-0xffffffffb3260fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045000] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052033] ... event mask: 000000070000000f [ 0.053656] rcu: Hierarchical SRCU implementation. [ 0.056504] smp: Bringing up secondary CPUs ... [ 0.057985] x86: Booting SMP configuration: [ 0.058058] .... node #0, CPUs: #1 #2 #3 [ 0.062224] smp: Brought up 1 node, 4 CPUs [ 0.063948] smpboot: Max logical packages: 1 [ 0.064043] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150357] node 0 deferred pages initialised in 82ms [ 0.154272] devtmpfs: initialized [ 0.157052] x86/mm: Memory block size: 128MB [ 0.160165] gcov: version magic: 0x41383552 [ 0.163202] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165271] pinctrl core: initialized pinctrl subsystem [ 0.166204] [ 0.166776] ************************************************************* [ 0.167017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168016] ** ** [ 0.169012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170012] ** ** [ 0.171016] ** This means that this kernel is built to expose internal ** [ 0.172015] ** IOMMU data structures, which may compromise security on ** [ 0.173014] ** your system. ** [ 0.174014] ** ** [ 0.175013] ** If you see this message and you are not debugging the ** [ 0.176016] ** kernel, report this immediately to your vendor! ** [ 0.177020] ** ** [ 0.178016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179014] ************************************************************* [ 0.180780] NET: Registered protocol family 16 [ 0.181486] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.185016] cpuidle: using governor menu [ 0.187753] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188500] PCI: Using configuration type 1 for base access [ 0.189134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.199080] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.200104] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.202094] cryptd: max_cpu_qlen set to 1000 [ 0.204484] ACPI: Added _OSI(Module Device) [ 0.205013] ACPI: Added _OSI(Processor Device) [ 0.206014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211436] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.214657] ACPI: Interpreter enabled [ 0.215060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.216014] ACPI: Using IOAPIC for interrupt routing [ 0.217103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.218389] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228964] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.229035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.230022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.233711] acpiphp: Slot [2] registered [ 0.234114] acpiphp: Slot [5] registered [ 0.235115] acpiphp: Slot [6] registered [ 0.236121] acpiphp: Slot [7] registered [ 0.237114] acpiphp: Slot [8] registered [ 0.238377] acpiphp: Slot [9] registered [ 0.239122] acpiphp: Slot [10] registered [ 0.240108] acpiphp: Slot [3] registered [ 0.241267] acpiphp: Slot [4] registered [ 0.242110] acpiphp: Slot [11] registered [ 0.243099] acpiphp: Slot [12] registered [ 0.244333] acpiphp: Slot [13] registered [ 0.245092] acpiphp: Slot [14] registered [ 0.246090] acpiphp: Slot [15] registered [ 0.247102] acpiphp: Slot [16] registered [ 0.248098] acpiphp: Slot [17] registered [ 0.249102] acpiphp: Slot [18] registered [ 0.250113] acpiphp: Slot [19] registered [ 0.251093] acpiphp: Slot [20] registered [ 0.252084] acpiphp: Slot [21] registered [ 0.253070] acpiphp: Slot [22] registered [ 0.254093] acpiphp: Slot [23] registered [ 0.255080] acpiphp: Slot [24] registered [ 0.256279] acpiphp: Slot [25] registered [ 0.257100] acpiphp: Slot [26] registered [ 0.258099] acpiphp: Slot [27] registered [ 0.259094] acpiphp: Slot [28] registered [ 0.260096] acpiphp: Slot [29] registered [ 0.261092] acpiphp: Slot [30] registered [ 0.262099] acpiphp: Slot [31] registered [ 0.263054] PCI host bridge to bus 0000:00 [ 0.264020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.265030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.266023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.267035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.268024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.269029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.270178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.272499] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.274516] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.280782] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.283269] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.284015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.285018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.286025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.287571] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.288865] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.289050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.290793] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.292941] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.300018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.303014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.308438] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.311017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.314012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.321017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.326670] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.329016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.332014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.340017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.347090] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.350019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.353034] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.361033] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.366832] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.369021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.372018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.380019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.387528] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.390019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.393017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.400064] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.407190] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.410023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.413019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.420022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.427790] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.428834] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.429482] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.430532] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.431497] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.435136] iommu: Default domain type: Passthrough [ 0.437999] SCSI subsystem initialized [ 0.440146] ACPI: bus type USB registered [ 0.443148] usbcore: registered new interface driver usbfs [ 0.447090] usbcore: registered new interface driver hub [ 0.450299] usbcore: registered new device driver usb [ 0.454198] pps_core: LinuxPPS API ver. 1 registered [ 0.456011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.462064] PTP clock support registered [ 0.468044] EDAC MC: Ver: 3.0.0 [ 0.470113] PCI: Using ACPI for IRQ routing [ 0.474200] NetLabel: Initializing [ 0.475013] NetLabel: domain hash size = 128 [ 0.477012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.479091] NetLabel: unlabeled traffic allowed by default [ 0.483045] vgaarb: loaded [ 0.485325] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.487257] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.495648] clocksource: Switched to clocksource kvm-clock [ 0.645695] VFS: Disk quotas dquot_6.6.0 [ 0.647789] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.651137] *** VALIDATE ramfs *** [ 0.652848] *** VALIDATE hugetlbfs *** [ 0.654230] pnp: PnP ACPI init [ 0.657707] pnp: PnP ACPI: found 6 devices [ 0.677001] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.680555] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.684079] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.688041] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.691969] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.695319] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.699784] NET: Registered protocol family 2 [ 0.703331] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.711678] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.715962] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.721603] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.725154] TCP: Hash tables configured (established 65536 bind 65536) [ 0.728297] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.731548] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.734919] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.738903] NET: Registered protocol family 1 [ 0.741995] RPC: Registered named UNIX socket transport module. [ 0.744702] RPC: Registered udp transport module. [ 0.746579] RPC: Registered tcp transport module. [ 0.748257] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.750810] NET: Registered protocol family 44 [ 0.752212] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.754319] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.756627] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.758705] PCI: CLS 0 bytes, default 64 [ 0.760487] Unpacking initramfs... [ 2.497832] debug: unmapping init [mem 0xffff8ff73cc54000-0xffff8ff73ffbffff] [ 2.503445] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.506345] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.510012] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.116627] Initialise system trusted keyrings [ 3.118387] Key type blacklist registered [ 3.120206] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.128078] zbud: loaded [ 3.131073] *** VALIDATE nfs *** [ 3.131884] *** VALIDATE nfs4 *** [ 3.133753] pstore: using deflate compression [ 3.137262] Platform Keyring initialized [ 3.275150] NET: Registered protocol family 38 [ 3.278033] Key type asymmetric registered [ 3.280674] Asymmetric key parser 'x509' registered [ 3.283334] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.287627] io scheduler mq-deadline registered [ 3.289568] io scheduler kyber registered [ 3.291564] io scheduler bfq registered [ 3.293092] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.295412] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.298373] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.301068] ACPI: Power Button [PWRF] [ 3.307389] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.314090] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.329614] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.338075] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.361055] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.389644] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.419436] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.424020] Non-volatile memory driver v1.3 [ 3.426546] Linux agpgart interface v0.103 [ 3.469354] virtio_blk virtio1: [vda] 134608 512-byte logical blocks (68.9 MB/65.7 MiB) [ 3.472636] vda: detected capacity change from 0 to 68919296 [ 3.495586] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.498754] vdb: detected capacity change from 0 to 1073741824 [ 3.521508] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.524172] vdc: detected capacity change from 0 to 2621440000 [ 3.546386] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.550251] vdd: detected capacity change from 0 to 2621440000 [ 3.566662] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.570357] vde: detected capacity change from 0 to 4294967296 [ 3.593181] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.596259] vdf: detected capacity change from 0 to 4294967296 [ 3.607133] libphy: Fixed MDIO Bus: probed [ 3.620037] usbcore: registered new interface driver usbserial_generic [ 3.622278] usbserial: USB Serial support registered for generic [ 3.624248] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.629104] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.631024] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.633609] mousedev: PS/2 mouse device common for all mice [ 3.636710] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.638242] rtc_cmos 00:05: RTC can wake from S4 [ 3.650040] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.654967] rtc_cmos 00:05: registered as rtc0 [ 3.659193] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.659301] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.665400] intel_pstate: CPU model not supported [ 3.668434] hid: raw HID events driver (C) Jiri Kosina [ 3.671169] usbcore: registered new interface driver usbhid [ 3.673551] usbhid: USB HID core driver [ 3.675635] drop_monitor: Initializing network drop monitor service [ 3.678326] Initializing XFRM netlink socket [ 3.680672] NET: Registered protocol family 10 [ 3.684430] Segment Routing with IPv6 [ 3.685654] NET: Registered protocol family 17 [ 3.687229] mpls_gso: MPLS GSO support [ 3.693740] RAS: Correctable Errors collector initialized. [ 3.696347] AVX version of gcm_enc/dec engaged. [ 3.698096] AES CTR mode by8 optimization enabled [ 3.788677] sched_clock: Marking stable (3788361163, 0)->(5124061090, -1335699927) [ 3.793395] registered taskstats version 1 [ 3.795563] Loading compiled-in X.509 certificates [ 3.797868] zswap: loaded using pool lzo/zbud [ 3.826527] Key type big_key registered [ 3.845543] Key type encrypted registered [ 3.847870] ima: No TPM chip found, activating TPM-bypass! [ 3.854735] ima: Allocated hash algorithm: sha1 [ 3.856887] ima: No architecture policies found [ 3.859364] evm: Initialising EVM extended attributes: [ 3.862579] evm: security.selinux [ 3.864315] evm: security.ima [ 3.865480] evm: security.capability [ 3.867269] evm: HMAC attrs: 0x1 [ 3.869713] rtc_cmos 00:05: setting system clock to 2026-04-29 20:17:54 UTC (1777493874) [ 3.877167] debug: unmapping init [mem 0xffffffffb4203000-0xffffffffb43fffff] [ 3.880409] debug: unmapping init [mem 0xffffffffb2f82000-0xffffffffb3258fff] [ 3.893147] Write protecting the kernel read-only data: 28672k [ 3.897154] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 3.900398] debug: unmapping init [mem 0xffffffffb1f14000-0xffffffffb1ffffff] [ 3.944216] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.952124] systemd[1]: Detected virtualization kvm. [ 3.954804] systemd[1]: Detected architecture x86-64. [ 3.957255] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.986674] systemd[1]: No hostname configured. [ 3.989213] systemd[1]: Set hostname to . [ 3.993030] random: systemd: uninitialized urandom read (16 bytes read) [ 3.997223] systemd[1]: Initializing machine ID from random generator. [ 4.218152] random: systemd: uninitialized urandom read (16 bytes read) [ 4.222606] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.227142] random: systemd: uninitialized urandom read (16 bytes read) [ 4.230705] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.237057] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.123191] device-mapper: uevent: version 1.0.3 [ 5.125622] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.064931] random: fast init done [ 6.064967] virtio_net virtio0 ens2: renamed from eth0 [ 6.241072] scsi host0: ata_piix [ 6.278631] scsi host1: ata_piix [ 6.298523] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.300654] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.365629] dracut-initqueue[582]: RTNETLINK answers: File exists [ 11.383302] random: crng init done [ 11.386219] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.011019] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.420648] printk: systemd: 19 output lines suppressed due to ratelimiting [ 13.741747] SELinux: Disabled at runtime. [ 13.810963] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.818353] systemd[1]: Detected virtualization kvm. [ 13.819571] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.456478] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.461689] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.468339] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.473791] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.479274] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.490754] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.495919] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ 14.752831] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ 15.081168] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.475633] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.541158] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.656582] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.672417] EDAC sbridge: Ver: 1.1.2 [ 18.077245] Key type dns_resolver registered [ 18.491229] NFS: Registering the id_resolver key type [ 18.493110] Key type id_resolver registered [ 18.494708] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg119-server login: [ 48.619544] libcfs: loading out-of-tree module taints kernel. [ 48.658571] Key type ._llcrypt registered [ 48.662296] Key type .llcrypt registered [ 48.778185] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 69.129514] hrtimer: interrupt took 6710194 ns [ 73.715390] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 77.031710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 77.061953] alg: No test for adler32 (adler32-zlib) [ 78.807827] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 80.391499] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 82.503290] Key type lgssc registered [ 86.460399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.280586] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 176.509125] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 198.302930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.347800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.840954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.918359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 200.166160] Lustre: lustre-MDT0000: new disk, initializing [ 200.418295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.495381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 206.462929] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.089888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.257861] Lustre: 6512:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 224.304867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 224.314143] Lustre: Skipped 1 previous similar message [ 224.454815] Lustre: lustre-MDT0001: new disk, initializing [ 224.571794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 224.645793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 224.665356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 230.731940] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.055134] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 250.772436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.185799] Lustre: lustre-OST0000: new disk, initializing [ 251.190929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 251.304066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.931230] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.258723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 261.278719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 261.431197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 278.677904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 278.818394] Lustre: lustre-OST0001: new disk, initializing [ 278.824314] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 278.921905] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 288.047952] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 288.841690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 288.858798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 288.958320] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 303.203391] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 313.836080] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 323.161390] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing check_logdir /tmp/testlogs/ [ 332.994592] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing yml_node [ 342.629914] Lustre: DEBUG MARKER: Client: 2.17.52.93 [ 348.187357] Lustre: DEBUG MARKER: MDS: 2.17.52.93 [ 353.063309] Lustre: DEBUG MARKER: OSS: 2.17.52.93 [ 355.526228] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Apr 29 16:23:44 EDT 2026 [ 384.450911] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 386.345357] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 403.943964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 403.963861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.010743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 406.496917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 406.503059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 406.551450] Lustre: Skipped 3 previous similar messages [ 411.617418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 411.622458] Lustre: Skipped 3 previous similar messages [ 416.737074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 416.751206] Lustre: Skipped 3 previous similar messages [ 417.249281] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 417.825237] Lustre: server umount lustre-MDT0000 complete [ 424.107804] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777494294 with bad export cookie 498327926506547543 [ 424.116689] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 424.135227] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 426.977495] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.981415] LustreError: 10637:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 427.018267] Lustre: Skipped 2 previous similar messages [ 427.019112] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 427.095375] LustreError: 10637:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 429.987632] LustreError: 6516:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.017564] LustreError: 6516:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 430.864895] Lustre: server umount lustre-MDT0001 complete [ 436.877269] Lustre: server umount lustre-OST0000 complete [ 442.138290] Lustre: server umount lustre-OST0001 complete [ 452.298414] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 464.730739] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 516.445364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.189980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 517.307179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 517.669060] Lustre: lustre-MDT0000: new disk, initializing [ 517.951845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.992901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 525.053751] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 541.452367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.783605] Lustre: 15001:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 541.923397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 541.936585] Lustre: Skipped 1 previous similar message [ 542.094413] Lustre: lustre-MDT0001: new disk, initializing [ 542.377083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 542.501876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 542.533894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 549.470878] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 560.794600] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 566.362077] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 576.984571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 577.633250] Lustre: lustre-OST0000: new disk, initializing [ 577.641934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 577.798729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 578.911633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 578.943284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 579.231655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 586.818535] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 597.609760] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 604.213764] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 604.607569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 611.568256] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 611.911575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 614.881097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 614.890427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.902061] LustreError: Skipped 1 previous similar message [ 614.925565] Lustre: Skipped 1 previous similar message [ 614.937932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 614.961514] Lustre: Skipped 4 previous similar messages [ 621.171319] Lustre: server umount lustre-OST0000 complete [ 629.226384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 629.243685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.285835] Lustre: Skipped 1 previous similar message [ 635.363837] LustreError: 15006:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.396364] LustreError: 15006:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 635.430994] Lustre: server umount lustre-MDT0000 complete [ 640.487891] LustreError: 15827:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.516802] LustreError: 15827:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 641.282072] LustreError: 14993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777494511 with bad export cookie 498327926506549076 [ 641.293462] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.298893] LustreError: 14993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 641.830922] Lustre: server umount lustre-MDT0001 complete [ 647.724750] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 16:28:36 (1777494516) [ 658.254679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.199287] LustreError: 18866:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.392119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.828327] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 668.642134] LustreError: 18866:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.672553] LustreError: 18866:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 679.267551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.628931] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 695.970786] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 702.046318] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 713.483612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 722.335931] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 732.081787] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 743.905807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 743.926393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 743.943855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 743.944988] Lustre: Skipped 2 previous similar messages [ 743.965244] Lustre: Skipped 6 previous similar messages [ 745.799222] Lustre: server umount lustre-OST0000 complete [ 751.587263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 757.715123] Lustre: server umount lustre-MDT0000 complete [ 759.276695] LustreError: 18866:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 759.300761] LustreError: 18866:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 762.749512] LustreError: 18846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777494633 with bad export cookie 498327926506549958 [ 762.761401] LustreError: 3656:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8ff682f28000 x1863837497372288/t0(0) o41->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 0 dl 1777494638 ref 1 fl Rpc:RESQU/200/0 rc -5/-107 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 762.765773] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 763.274743] Lustre: server umount lustre-MDT0001 complete [ 779.260787] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 783.945428] Key type lgssc unregistered [ 784.503730] LNet: 22051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.528351] LNetError: 22051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.558400] LNet: Removed LNI 192.168.201.119@tcp [ 785.844193] Key type .llcrypt unregistered [ 785.846271] Key type ._llcrypt unregistered [ 819.867493] Key type ._llcrypt registered [ 819.870823] Key type .llcrypt registered [ 820.117517] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 822.185020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 822.366481] alg: No test for adler32 (adler32-zlib) [ 823.928472] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 824.305286] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 826.015327] Key type lgssc registered [ 827.561675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 836.175256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 836.190639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.816812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 844.729803] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 848.356664] LustreError: 23446:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.428935] LustreError: 23446:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 853.479830] LustreError: 23447:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 856.559458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.201685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 863.776775] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 872.526922] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 877.501478] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 887.465498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 888.243733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.831388] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 905.686828] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 907.782067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 932.321865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 932.338571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.367682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.356316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.369550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 933.394442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 936.173764] Lustre: server umount lustre-OST0000 complete [ 943.589292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.602971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.625508] Lustre: Skipped 1 previous similar message [ 943.643072] Lustre: Skipped 2 previous similar messages [ 948.128631] Lustre: server umount lustre-MDT0000 complete [ 948.704758] LustreError: 23447:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.733174] LustreError: 23447:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 953.445504] LustreError: 23428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777494824 with bad export cookie 1620389400347464684 [ 953.474768] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.087823] Lustre: server umount lustre-MDT0001 complete [ 970.270334] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 975.001708] Key type lgssc unregistered [ 975.538830] LNet: 27073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.556270] LNetError: 27073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.584784] LNet: Removed LNI 192.168.201.119@tcp [ 976.842306] Key type .llcrypt unregistered [ 976.854389] Key type ._llcrypt unregistered [ 1012.710151] Key type ._llcrypt registered [ 1012.714340] Key type .llcrypt registered [ 1012.887502] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1015.290673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.328577] alg: No test for adler32 (adler32-zlib) [ 1016.856903] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1017.293692] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1019.231191] Key type lgssc registered [ 1021.839367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1034.599841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1034.630442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.731956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.169583] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1047.013065] LustreError: 28477:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1047.033900] LustreError: 28477:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1052.134385] LustreError: 28476:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1057.189490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.248579] LustreError: 28477:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1057.708799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1063.777481] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1073.110115] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1079.999857] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1098.845596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1099.392581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1106.559585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1109.239769] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1116.502766] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1127.408879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.421222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1127.437248] Lustre: Skipped 1 previous similar message [ 1127.437654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1127.461634] Lustre: Skipped 1 previous similar message [ 1132.517617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1132.530412] Lustre: Skipped 1 previous similar message [ 1137.645574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.664804] Lustre: Skipped 1 previous similar message [ 1141.215644] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1141.404832] Lustre: server umount lustre-OST0000 complete [ 1147.877485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.884091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.899990] Lustre: Skipped 1 previous similar message [ 1147.935352] Lustre: Skipped 1 previous similar message [ 1152.992741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.006864] Lustre: Skipped 1 previous similar message [ 1153.635283] Lustre: server umount lustre-MDT0000 complete [ 1158.124784] LustreError: 29540:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.168504] LustreError: 29540:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1159.262669] LustreError: 28456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777495029 with bad export cookie 2209279313248595041 [ 1159.268396] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.285215] LustreError: 28456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1160.224567] Lustre: server umount lustre-MDT0001 complete [ 1177.309523] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1182.031468] Key type lgssc unregistered [ 1182.518453] LNet: 32005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.528587] LNetError: 32005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.589349] LNet: Removed LNI 192.168.201.119@tcp [ 1185.219327] Key type .llcrypt unregistered [ 1185.226646] Key type ._llcrypt unregistered [ 1218.770905] Key type ._llcrypt registered [ 1218.774198] Key type .llcrypt registered [ 1218.907836] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1220.493439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1220.699050] alg: No test for adler32 (adler32-zlib) [ 1222.050736] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1222.374924] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1224.304672] Key type lgssc registered [ 1226.390239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1237.772909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1237.808685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.511219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1245.767719] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1249.774963] LustreError: 33407:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.816709] LustreError: 33407:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1254.914226] LustreError: 33408:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.244530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.990194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1267.200671] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1274.733164] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1279.995527] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1289.418342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1290.074653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.424552] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1306.590534] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1309.588032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1325.030470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.039715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1325.089820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1325.102297] LustreError: Skipped 1 previous similar message [ 1325.119422] Lustre: Skipped 1 previous similar message [ 1329.041764] Lustre: server umount lustre-OST0000 complete [ 1335.265688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.283598] Lustre: Skipped 1 previous similar message [ 1335.288077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1340.215967] Lustre: server umount lustre-MDT0000 complete [ 1340.385616] LustreError: 33408:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.429398] LustreError: 33408:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1344.491370] LustreError: 33402:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.524979] LustreError: 33402:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1346.218104] LustreError: 33387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777495216 with bad export cookie 668463926242410090 [ 1346.235837] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1346.247884] LustreError: 33387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1346.846707] Lustre: server umount lustre-MDT0001 complete [ 1363.564713] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1368.028842] Key type lgssc unregistered [ 1368.500439] LNet: 36609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.521569] LNetError: 36609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1368.560066] LNet: Removed LNI 192.168.201.119@tcp [ 1369.802947] Key type .llcrypt unregistered [ 1369.807629] Key type ._llcrypt unregistered [ 1405.924629] Key type ._llcrypt registered [ 1405.927756] Key type .llcrypt registered [ 1406.233164] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1407.583254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1407.607284] alg: No test for adler32 (adler32-zlib) [ 1409.318570] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1409.532767] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1411.207176] Key type lgssc registered [ 1414.019623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1426.213517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1426.252922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.804245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.884450] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1438.179552] LustreError: 38013:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.213326] LustreError: 38013:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1439.226882] LustreError: 38014:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1444.324773] LustreError: 38013:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.916538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1447.621374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1453.775158] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1462.266825] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1468.581927] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1477.255608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.912948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.620792] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1494.140698] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1498.431379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1506.237882] Lustre: lustre-OST0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 1508.834705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1508.843342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1508.865865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1508.872249] Lustre: Skipped 1 previous similar message [ 1511.328477] Lustre: lustre-OST0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 1512.688449] Lustre: server umount lustre-OST0000 complete [ 1539.555251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1539.557932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.579153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.602196] Lustre: Skipped 2 previous similar messages [ 1543.892309] Lustre: server umount lustre-MDT0000 complete [ 1544.701892] LustreError: 38014:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.721221] LustreError: 38014:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1548.840861] LustreError: 38924:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.886803] LustreError: 38924:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1550.098822] LustreError: 39633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777495420 with bad export cookie 5484262710729636003 [ 1550.122980] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1556.980294] Lustre: server umount lustre-MDT0001 complete [ 1573.877659] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1578.529727] Key type lgssc unregistered [ 1579.051975] LNet: 41370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.067312] LNetError: 41370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.092929] LNet: Removed LNI 192.168.201.119@tcp [ 1580.581970] Key type .llcrypt unregistered [ 1580.585596] Key type ._llcrypt unregistered [ 1616.269371] Key type ._llcrypt registered [ 1616.274331] Key type .llcrypt registered [ 1616.511736] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1619.328060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1619.386790] alg: No test for adler32 (adler32-zlib) [ 1620.893534] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1621.303404] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1623.103409] Key type lgssc registered [ 1625.479303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1637.007390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1637.065394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.755499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.753735] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1649.128076] LustreError: 42768:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1649.157468] LustreError: 42768:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1654.256573] LustreError: 42767:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.701729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.120765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1663.271120] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1671.298448] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1676.966946] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1686.820765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1687.502082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.108901] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1709.073944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1710.984907] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1723.872676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1723.885269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.914821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1724.388394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.420672] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 1724.460595] Lustre: Skipped 2 previous similar messages [ 1726.443135] Lustre: server umount lustre-MDT0000 complete [ 1729.451059] LustreError: 42764:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1732.940949] LustreError: 42747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777495603 with bad export cookie 7187671832193987780 [ 1732.956975] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1732.966461] LustreError: 42747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1734.567916] LustreError: 42764:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1734.586420] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.19@tcp (stopping) [ 1734.597440] LustreError: 42764:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1734.621904] Lustre: Skipped 1 previous similar message [ 1734.638855] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.652714] Lustre: Skipped 1 previous similar message [ 1737.636478] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1737.660054] Lustre: Skipped 1 previous similar message [ 1739.847234] Lustre: server umount lustre-MDT0001 complete [ 1757.245288] Lustre: server umount lustre-OST0000 complete [ 1784.416812] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 1790.406061] Key type lgssc unregistered [ 1790.937211] LNet: 46297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.949888] LNetError: 46297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.990702] LNet: Removed LNI 192.168.201.119@tcp [ 1792.633909] Key type .llcrypt unregistered [ 1792.635598] Key type ._llcrypt unregistered [ 1828.134870] Key type ._llcrypt registered [ 1828.140938] Key type .llcrypt registered [ 1828.326406] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 1830.762757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1830.901939] alg: No test for adler32 (adler32-zlib) [ 1832.396486] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 1832.873020] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 1834.711626] Key type lgssc registered [ 1836.733571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1849.138393] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1849.162684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1864.932145] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1870.047336] LustreError: 47669:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 1896.800206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.279102] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 1915.829464] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2083.714915] Lustre: server umount lustre-OST0000 complete [ 2108.222704] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2112.437756] Key type lgssc unregistered [ 2112.920173] LNet: 49219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.959556] LNetError: 49219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.982856] LNet: Removed LNI 192.168.201.119@tcp [ 2114.331669] Key type .llcrypt unregistered [ 2114.339578] Key type ._llcrypt unregistered [ 2149.887847] Key type ._llcrypt registered [ 2149.892771] Key type .llcrypt registered [ 2150.089617] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2152.219801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2152.296966] alg: No test for adler32 (adler32-zlib) [ 2153.541434] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2153.948246] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2155.778802] Key type lgssc registered [ 2157.837653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2171.550644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2171.603298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.476651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.364995] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2183.667981] LustreError: 50621:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.712416] LustreError: 50621:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2188.773651] LustreError: 50622:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.912987] LustreError: 50621:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.313423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.071645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2202.041732] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2211.577221] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2218.631505] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2229.581234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2230.469492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.461602] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2243.844036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2248.983072] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2254.331349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.341585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2254.348669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2254.359108] Lustre: Skipped 1 previous similar message [ 2259.426952] Lustre: server umount lustre-OST0000 complete [ 2266.593717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2266.605177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.632215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2266.651200] Lustre: Skipped 1 previous similar message [ 2269.664656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.672975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.917868] Lustre: server umount lustre-MDT0000 complete [ 2274.783972] LustreError: 50621:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.833870] LustreError: 50621:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2277.502422] LustreError: 50600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777496148 with bad export cookie 13015441433524126745 [ 2277.508348] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.516773] LustreError: 50600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2278.052507] Lustre: server umount lustre-MDT0001 complete [ 2293.773869] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2298.414666] Key type lgssc unregistered [ 2298.850623] LNet: 53799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.857246] LNetError: 53799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2298.885592] LNet: Removed LNI 192.168.201.119@tcp [ 2300.739795] Key type .llcrypt unregistered [ 2300.744295] Key type ._llcrypt unregistered [ 2338.340085] Key type ._llcrypt registered [ 2338.343499] Key type .llcrypt registered [ 2338.560337] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2340.240706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2340.366943] alg: No test for adler32 (adler32-zlib) [ 2341.630731] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2341.966087] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2343.671225] Key type lgssc registered [ 2345.430141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2357.740717] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2357.773069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2373.602947] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2378.719306] LustreError: 55169:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 2405.471360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.581510] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2425.188476] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2436.242771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.085610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.243294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2444.191487] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2445.297886] LustreError: 56097:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.340547] LustreError: 56097:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 2450.405952] LustreError: 56096:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.458611] LustreError: 56096:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2455.527572] LustreError: 55194:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.555167] LustreError: 55194:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2458.430729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.954111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2466.508267] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2475.897556] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2481.699224] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2488.289171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2488.309749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.323329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.826697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.835833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2489.843831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.641274] Lustre: server umount lustre-OST0000 complete [ 2519.534531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.538414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2519.561359] Lustre: Skipped 1 previous similar message [ 2519.574964] Lustre: Skipped 2 previous similar messages [ 2524.025362] Lustre: server umount lustre-MDT0000 complete [ 2524.640388] LustreError: 56096:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.664915] LustreError: 56096:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2529.706524] LustreError: 55175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777496400 with bad export cookie 14473121917358459993 [ 2529.726713] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.735145] LustreError: 55175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2536.686848] Lustre: server umount lustre-MDT0001 complete [ 2554.423178] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2559.254945] Key type lgssc unregistered [ 2559.846678] LNet: 58568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2559.856753] LNetError: 58568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2559.880365] LNet: Removed LNI 192.168.201.119@tcp [ 2561.330779] Key type .llcrypt unregistered [ 2561.334480] Key type ._llcrypt unregistered [ 2597.607910] Key type ._llcrypt registered [ 2597.612086] Key type .llcrypt registered [ 2597.823372] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2599.885193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2600.252838] alg: No test for adler32 (adler32-zlib) [ 2601.659423] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2602.087792] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2603.920887] Key type lgssc registered [ 2605.897143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2618.474934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2618.525272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.623979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2627.737389] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2631.154824] LustreError: 59981:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.195115] LustreError: 59981:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2636.257663] LustreError: 59982:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.916830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.418501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2648.208833] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2656.395094] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2662.116650] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2673.004779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2673.768996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.319702] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2691.249189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2694.152348] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2727.393383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2727.405747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.418833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2727.429386] Lustre: Skipped 1 previous similar message [ 2727.458453] Lustre: Skipped 1 previous similar message [ 2731.899263] Lustre: server umount lustre-OST0000 complete [ 2737.632871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.664111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2737.681155] Lustre: Skipped 1 previous similar message [ 2742.753870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2742.772474] Lustre: Skipped 1 previous similar message [ 2743.899346] Lustre: server umount lustre-MDT0000 complete [ 2747.878900] LustreError: 60693:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.906717] LustreError: 60693:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2749.177489] LustreError: 59963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777496619 with bad export cookie 12398958391318125673 [ 2749.181358] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2749.205143] LustreError: 59963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2749.854143] Lustre: server umount lustre-MDT0001 complete [ 2769.578750] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 2775.399371] Key type lgssc unregistered [ 2776.157495] LNet: 63212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2776.179263] LNetError: 63212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2776.200350] LNet: Removed LNI 192.168.201.119@tcp [ 2777.473807] Key type .llcrypt unregistered [ 2777.478816] Key type ._llcrypt unregistered [ 2815.472706] Key type ._llcrypt registered [ 2815.485426] Key type .llcrypt registered [ 2815.693086] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 2817.593241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2817.767536] alg: No test for adler32 (adler32-zlib) [ 2819.234465] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2819.629619] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 2821.431186] Key type lgssc registered [ 2823.680795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2838.217947] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 17:05:06 (1777496706) [ 2847.956846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2847.977494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2849.829562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.777889] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2860.001490] LustreError: 64919:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.020611] LustreError: 64919:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2865.120166] LustreError: 64920:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2870.012335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.241210] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2870.627843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2877.538247] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2886.700664] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2893.165180] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2904.040717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2904.621330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.941884] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2920.013795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2923.536127] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2930.727341] Lustre: Failing over lustre-MDT0000 [ 2931.239811] Lustre: server umount lustre-MDT0000 complete [ 2932.193172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2932.215976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.250833] LustreError: 64919:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.282052] LustreError: 66424:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.283913] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.360262] Lustre: Skipped 1 previous similar message [ 2940.398326] LustreError: 64920:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.437211] LustreError: 64920:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 2945.173089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.434754] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2945.692629] Lustre: *** cfs_fail_loc=135, val=0*** [ 2945.933886] LustreError: 67669:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2945.951344] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (not set up) [ 2945.967826] LustreError: 67669:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.119@tcp: cfg command failed: rc = -2 [ 2945.967892] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2945.967892] [ 2945.975238] LustreError: MGC192.168.201.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2946.062551] LustreError: 67657:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2946.070922] LustreError: 67657:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2946.093038] LustreError: 67657:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2946.101674] LustreError: 67657:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2946.146859] Lustre: server umount lustre-MDT0000 complete [ 2946.152817] LustreError: 67657:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2950.629656] LustreError: 64920:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.654889] LustreError: 64920:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 2956.204936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.433223] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2956.452521] LustreError: 64911:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2956.930654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2957.063526] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2961.306495] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2962.419725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2962.458386] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2962.546056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2963.784455] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 2977.258554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2977.266052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.288348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2982.882041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2982.918446] Lustre: Skipped 1 previous similar message [ 2988.004375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2988.016052] Lustre: Skipped 1 previous similar message [ 2990.049712] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2990.313364] Lustre: server umount lustre-OST0000 complete [ 2996.194978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2996.216677] Lustre: Skipped 1 previous similar message [ 2996.226053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2996.243584] Lustre: Skipped 2 previous similar messages [ 3002.478732] Lustre: server umount lustre-MDT0000 complete [ 3006.440636] LustreError: 64919:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.485655] LustreError: 64919:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 3007.906349] LustreError: 64900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777496878 with bad export cookie 11696302861306882976 [ 3007.909058] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.917862] LustreError: 64900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3008.529841] Lustre: server umount lustre-MDT0001 complete [ 3024.569349] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3028.674393] Key type lgssc unregistered [ 3029.044219] LNet: 69418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3029.059442] LNetError: 69418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3029.085672] LNet: Removed LNI 192.168.201.119@tcp [ 3030.270816] Key type .llcrypt unregistered [ 3030.273812] Key type ._llcrypt unregistered [ 3063.068344] Key type ._llcrypt registered [ 3063.072994] Key type .llcrypt registered [ 3063.260185] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3064.927587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3065.037617] alg: No test for adler32 (adler32-zlib) [ 3066.467569] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 3066.803535] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3068.584087] Key type lgssc registered [ 3070.667773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3082.684274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3082.776704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.820844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.685336] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3095.012478] LustreError: 70831:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.031278] LustreError: 70831:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3100.133753] LustreError: 70830:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3103.447724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.863368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3110.607847] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3119.603955] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3125.294234] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3135.158408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3136.053668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.372964] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3154.174909] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3155.525619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 3162.224414] Lustre: Failing over lustre-MDT0000 [ 3162.664518] Lustre: server umount lustre-MDT0000 complete [ 3165.154418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3165.176325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.219757] LustreError: 70831:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.683360] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.706533] Lustre: Skipped 1 previous similar message [ 3170.793962] LustreError: 70830:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.834030] LustreError: 70830:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 3175.753596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.906113] LustreError: 70830:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.944455] LustreError: 70830:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3176.046533] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3176.258855] Lustre: *** cfs_fail_loc=a05, val=0*** [ 3176.425494] LustreError: 73585:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 3176.429676] LustreError: 73585:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.119@tcp: cfg command failed: rc = -9 [ 3176.435295] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3176.435295] [ 3176.440413] LustreError: MGC192.168.201.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3176.448948] LustreError: 73572:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 3176.455093] LustreError: 73572:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 3176.478479] LustreError: 73572:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3176.489706] LustreError: 73572:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3176.543456] Lustre: server umount lustre-MDT0000 complete [ 3176.550170] LustreError: 73572:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 3185.557609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.758518] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3185.777504] LustreError: 70822:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3186.102883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.180708] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3187.631726] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3191.283494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3191.305502] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3191.332812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 3191.813607] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3205.602482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3205.617590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.657431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3206.630965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3206.656662] Lustre: Skipped 1 previous similar message [ 3209.702685] Lustre: server umount lustre-OST0000 complete [ 3216.867335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.870769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3216.898620] Lustre: Skipped 1 previous similar message [ 3216.902800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3221.651673] Lustre: server umount lustre-MDT0000 complete [ 3222.000195] LustreError: 70831:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3222.028915] LustreError: 70831:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 3227.249625] LustreError: 71549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497097 with bad export cookie 6014453111825599971 [ 3227.251389] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3227.271332] LustreError: 71549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3227.806889] Lustre: server umount lustre-MDT0001 complete [ 3246.231901] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3251.911409] Key type lgssc unregistered [ 3252.507392] LNet: 75334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3252.522275] LNetError: 75334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3252.541334] LNet: Removed LNI 192.168.201.119@tcp [ 3254.306228] Key type .llcrypt unregistered [ 3254.310777] Key type ._llcrypt unregistered [ 3273.888692] Key type ._llcrypt registered [ 3273.891734] Key type .llcrypt registered [ 3274.032404] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3274.634379] Key type .llcrypt unregistered [ 3274.641175] Key type ._llcrypt unregistered [ 3309.766480] Key type ._llcrypt registered [ 3309.771788] Key type .llcrypt registered [ 3309.962315] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3311.842759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3311.868808] alg: No test for adler32 (adler32-zlib) [ 3313.067619] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 3313.485081] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3315.343420] Key type lgssc registered [ 3317.092410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3330.402097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3330.438947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.603764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3339.615809] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3342.826491] LustreError: 77209:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.876748] LustreError: 77209:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3347.940667] LustreError: 77208:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.061637] LustreError: 77209:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3353.409401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.181649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3360.692823] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3368.483301] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3373.731105] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3383.618043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.312089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3395.285750] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3402.823642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 3403.699442] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3420.136745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3420.162340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.196776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3421.222168] Lustre: server umount lustre-OST0000 complete [ 3428.325548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3428.340163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3428.350056] Lustre: Skipped 1 previous similar message [ 3433.441319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3433.450142] Lustre: Skipped 1 previous similar message [ 3436.887897] Lustre: server umount lustre-MDT0000 complete [ 3437.538359] LustreError: 77209:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3437.566879] LustreError: 77209:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3442.660740] LustreError: 77208:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3442.727222] LustreError: 77208:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3443.280759] LustreError: 77190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497313 with bad export cookie 8384036977247268979 [ 3443.290651] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3443.302592] LustreError: 77190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3444.191101] Lustre: server umount lustre-MDT0001 complete [ 3463.613854] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3468.744139] Key type lgssc unregistered [ 3469.164957] LNet: 80428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3469.178854] LNetError: 80428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3469.195090] LNet: Removed LNI 192.168.201.119@tcp [ 3470.832467] Key type .llcrypt unregistered [ 3470.842320] Key type ._llcrypt unregistered [ 3506.461960] Key type ._llcrypt registered [ 3506.468736] Key type .llcrypt registered [ 3506.765190] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3508.539313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3508.626088] alg: No test for adler32 (adler32-zlib) [ 3509.956506] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 3510.474652] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3512.234066] Key type lgssc registered [ 3514.461731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3527.322399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3527.399941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3529.518097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3537.098940] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3539.941669] LustreError: 81828:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3539.976030] LustreError: 81828:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3545.070033] LustreError: 81827:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3550.176499] LustreError: 81828:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3551.011297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.610128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3558.260847] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3566.845935] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3572.582309] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3583.522550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3583.929822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.744828] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3600.324572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3603.422965] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3615.718502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.735571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3615.739693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3615.755033] Lustre: Skipped 1 previous similar message [ 3620.497528] Lustre: server umount lustre-OST0000 complete [ 3625.958401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.965335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3625.978197] Lustre: Skipped 2 previous similar messages [ 3625.998664] Lustre: Skipped 1 previous similar message [ 3631.072560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3631.090794] Lustre: Skipped 1 previous similar message [ 3636.197171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3636.203465] Lustre: Skipped 1 previous similar message [ 3640.287859] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3640.863411] Lustre: server umount lustre-MDT0000 complete [ 3641.316997] LustreError: 82539:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3641.356838] LustreError: 82539:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3646.435270] LustreError: 81827:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.455859] LustreError: 81827:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3647.825103] LustreError: 81808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497518 with bad export cookie 12968151138339698161 [ 3647.831670] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3647.859189] LustreError: 81808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3648.589330] Lustre: server umount lustre-MDT0001 complete [ 3666.596612] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3672.859577] Key type lgssc unregistered [ 3673.111682] LNet: 85011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3673.118119] LNetError: 85011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3673.131055] LNet: Removed LNI 192.168.201.119@tcp [ 3673.973748] Key type .llcrypt unregistered [ 3673.977777] Key type ._llcrypt unregistered [ 3710.211411] Key type ._llcrypt registered [ 3710.218804] Key type .llcrypt registered [ 3710.430094] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3712.781701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3712.831760] alg: No test for adler32 (adler32-zlib) [ 3713.932957] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 3714.409779] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3716.191785] Key type lgssc registered [ 3717.744180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3728.919688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3728.949509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.889886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3736.791823] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3741.164560] LustreError: 86407:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.186557] LustreError: 86407:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3746.273944] LustreError: 86406:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3749.436388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3750.086786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3756.583122] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3764.888680] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3770.661881] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3780.419736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3781.084055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.695909] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3800.548532] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3801.602949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3820.005458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3820.021851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.046410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3821.258337] Lustre: server umount lustre-OST0000 complete [ 3831.786801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3831.798914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.833641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3836.395628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3836.415779] Lustre: Skipped 3 previous similar messages [ 3837.746589] Lustre: server umount lustre-MDT0000 complete [ 3841.508139] LustreError: 87120:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3841.564383] LustreError: 87120:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3844.737982] LustreError: 86388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497715 with bad export cookie 2328786192975388383 [ 3844.749638] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3844.771427] LustreError: 86388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3845.451557] Lustre: server umount lustre-MDT0001 complete [ 3861.923682] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 3866.465989] Key type lgssc unregistered [ 3866.785390] LNet: 89611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3866.806553] LNetError: 89611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3867.887819] LNet: Removed LNI 192.168.201.119@tcp [ 3869.353419] Key type .llcrypt unregistered [ 3869.356966] Key type ._llcrypt unregistered [ 3903.617580] Key type ._llcrypt registered [ 3903.621458] Key type .llcrypt registered [ 3903.793350] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 3905.835895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3905.924769] alg: No test for adler32 (adler32-zlib) [ 3907.070368] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 3907.433176] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 3909.121047] Key type lgssc registered [ 3910.461771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3921.182113] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3921.211800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3936.863778] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3942.049976] LustreError: 90981:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 3968.735244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.722636] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 3985.493673] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3992.633291] Lustre: server umount lustre-OST0000 complete [ 4004.388194] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 17:24:33 (1777497873) [ 4013.979441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4014.689842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.682722] LustreError: 92517:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4019.704898] LustreError: 92517:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4021.512415] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4024.813753] LustreError: 92518:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.920743] LustreError: 92517:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4032.844037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4033.419435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4039.673947] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4049.030831] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4055.414072] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4065.300490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4065.806528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4074.385880] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4081.916554] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4085.268865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 4103.650354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4103.661521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4103.678972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4105.699987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.700462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4108.576735] Lustre: server umount lustre-OST0000 complete [ 4115.424403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4115.441220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.475524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4115.502396] Lustre: Skipped 1 previous similar message [ 4120.201979] Lustre: server umount lustre-MDT0000 complete [ 4121.060959] LustreError: 92517:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.490511] LustreError: 92498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777497996 with bad export cookie 7306112925806721275 [ 4125.530942] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4125.532888] LustreError: 92498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4126.463028] Lustre: server umount lustre-MDT0001 complete [ 4141.975423] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 4145.720723] Key type lgssc unregistered [ 4146.201074] LNet: 95881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4146.211904] LNetError: 95881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4146.227970] LNet: Removed LNI 192.168.201.119@tcp [ 4147.418497] Key type .llcrypt unregistered [ 4147.424324] Key type ._llcrypt unregistered [ 4205.821419] Key type ._llcrypt registered [ 4205.823677] Key type .llcrypt registered [ 4206.026201] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4207.902788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4207.951176] alg: No test for adler32 (adler32-zlib) [ 4209.050241] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 4209.256092] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 4211.040432] Key type lgssc registered [ 4212.636725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4223.826713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4223.852779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.381486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.511802] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4234.730729] LustreError: 98051:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.764225] LustreError: 98051:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4239.845876] LustreError: 98052:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4242.162451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.652880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4247.573710] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4254.865195] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4259.934922] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4269.263962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4269.552427] LustreError: 99646:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4269.861642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.190157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 4277.624926] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4285.231990] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4297.698049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4297.716741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4297.742272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4297.754991] Lustre: Skipped 1 previous similar message [ 4302.872417] Lustre: server umount lustre-OST0000 complete [ 4307.938033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.941839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4307.950620] Lustre: Skipped 2 previous similar messages [ 4307.960858] Lustre: Skipped 1 previous similar message [ 4313.064785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4313.075679] Lustre: Skipped 1 previous similar message [ 4318.179793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.188555] Lustre: Skipped 1 previous similar message [ 4322.273603] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4322.776239] Lustre: server umount lustre-MDT0000 complete [ 4323.297032] LustreError: 98051:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4323.314085] LustreError: 98051:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4327.801934] LustreError: 98033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498198 with bad export cookie 13284131953372700016 [ 4327.820967] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4327.832085] LustreError: 98033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4328.478748] Lustre: server umount lustre-MDT0001 complete [ 4343.606443] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 4347.875233] Key type lgssc unregistered [ 4348.224391] LNet: 101230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4348.237511] LNetError: 101230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4348.283367] LNet: Removed LNI 192.168.201.119@tcp [ 4349.363655] Key type .llcrypt unregistered [ 4349.367484] Key type ._llcrypt unregistered [ 4374.095631] Key type ._llcrypt registered [ 4374.098723] Key type .llcrypt registered [ 4374.308117] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4376.187732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4376.323394] alg: No test for adler32 (adler32-zlib) [ 4377.415077] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 4377.701652] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 4379.447531] Key type lgssc registered [ 4381.228433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4391.488956] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4391.531486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4407.136169] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4412.258437] LustreError: 102475:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 4439.008255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4447.475265] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4454.233989] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4462.070825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4462.461062] LustreError: MGC192.168.201.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4462.506405] LustreError: 103386:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4462.516453] LustreError: 103386:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 4462.524437] LustreError: 103386:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 4462.663520] Lustre: server umount lustre-MDT0000 complete [ 4462.667746] LustreError: 103386:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4483.039840] Lustre: 101811:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777498337/real 1777498337] req@00000000ba221918 x1863842002898304/t0(0) o400->MGC192.168.201.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777498353 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4483.078595] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4483.088557] LustreError: 102487:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4503.717326] Lustre: server umount lustre-OST0000 complete [ 4515.869206] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 4528.732351] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4572.135095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4572.487233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4572.511220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4572.615154] Lustre: lustre-MDT0000: new disk, initializing [ 4572.748046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4572.770428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4577.962882] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4588.913891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4589.088969] Lustre: 106148:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4589.205664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4589.213387] Lustre: Skipped 1 previous similar message [ 4589.315596] Lustre: lustre-MDT0001: new disk, initializing [ 4589.500348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4589.554981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4589.571116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4594.398861] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4601.967028] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4605.699920] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4612.818055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.049428] Lustre: lustre-OST0000: new disk, initializing [ 4613.053757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4613.178275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4620.021680] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4623.438233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4623.449248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4623.586244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4627.281643] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4631.969453] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4632.247497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4636.936852] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4637.198527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4644.325804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4644.339114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4644.341611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4644.344238] Lustre: Skipped 1 previous similar message [ 4644.358788] LustreError: Skipped 1 previous similar message [ 4645.666561] Lustre: server umount lustre-OST0000 complete [ 4650.983313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4650.991646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4651.019586] Lustre: Skipped 1 previous similar message [ 4651.025989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4654.561649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4654.562988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4655.932282] Lustre: server umount lustre-MDT0000 complete [ 4659.675468] LustreError: 106927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498530 with bad export cookie 13549775534297293604 [ 4659.677668] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4659.693161] LustreError: 106927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4660.132725] Lustre: server umount lustre-MDT0001 complete [ 4668.874636] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 17:35:38 (1777498538) [ 4672.583703] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4688.873893] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 4699.690898] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4738.934689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4739.219541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4739.251723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4739.321201] Lustre: lustre-MDT0000: new disk, initializing [ 4739.408250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4739.425225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4744.015567] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4755.392443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.502307] Lustre: 112612:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4755.561288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4755.570074] Lustre: Skipped 1 previous similar message [ 4755.670608] Lustre: lustre-MDT0001: new disk, initializing [ 4755.792359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4755.822763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4755.830668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4760.676136] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4769.548613] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4773.565359] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4780.623197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4780.912975] Lustre: lustre-OST0000: new disk, initializing [ 4780.917539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4781.015891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4787.272338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4787.285961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4787.416755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4787.713873] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4794.680574] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4799.486936] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4799.715943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4804.180180] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4804.394095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4808.180357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4808.186831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4808.208466] Lustre: Skipped 1 previous similar message [ 4808.217288] Lustre: Skipped 1 previous similar message [ 4808.223800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4812.756723] Lustre: server umount lustre-OST0000 complete [ 4817.376685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4817.390283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.425430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4823.089534] Lustre: server umount lustre-MDT0000 complete [ 4827.045413] LustreError: 112605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498697 with bad export cookie 13549775534297294479 [ 4827.055782] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4827.060293] LustreError: 112605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4827.305123] Lustre: server umount lustre-MDT0001 complete [ 4834.756024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.213484] LustreError: 116424:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.232130] LustreError: 116424:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4835.290272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.197050] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4840.420154] LustreError: 116425:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.528124] LustreError: 116424:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.654341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4852.177805] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4858.171387] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4862.302958] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4868.504315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4874.570195] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 4879.610227] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4889.507756] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4893.664338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4893.671104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.681453] Lustre: Skipped 1 previous similar message [ 4893.688831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4893.693295] Lustre: Skipped 2 previous similar messages [ 4899.022598] Lustre: server umount lustre-OST0000 complete [ 4908.014340] Lustre: server umount lustre-MDT0000 complete [ 4910.920661] LustreError: 116406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498781 with bad export cookie 13549775534297295368 [ 4910.923320] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4910.932021] LustreError: 116406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4911.192683] Lustre: server umount lustre-MDT0001 complete [ 4919.635607] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 4921.748769] Key type lgssc unregistered [ 4921.980571] LNet: 119696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4921.985322] LNetError: 119696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4921.993714] LNet: Removed LNI 192.168.201.119@tcp [ 4922.593133] Key type .llcrypt unregistered [ 4922.595216] Key type ._llcrypt unregistered [ 4936.345104] Key type ._llcrypt registered [ 4936.349249] Key type .llcrypt registered [ 4936.443753] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 4948.690847] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 4949.439761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4949.480376] alg: No test for adler32 (adler32-zlib) [ 4950.463226] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 4950.614659] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 4952.263159] Key type lgssc registered [ 4953.069991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4996.780246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4996.813270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.428071] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4998.481527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4998.548655] Lustre: lustre-MDT0000: new disk, initializing [ 4998.706145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.732704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5006.467790] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5028.021072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.331886] Lustre: 122794:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5028.570763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5028.580250] Lustre: Skipped 1 previous similar message [ 5028.934456] Lustre: lustre-MDT0001: new disk, initializing [ 5029.225354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5029.330828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5029.352334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5037.809973] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5050.395890] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5056.632579] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5066.416128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5066.833553] Lustre: lustre-OST0000: new disk, initializing [ 5066.839137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5066.995264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5075.047832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5075.070977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5075.300372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5076.015268] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5085.481842] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5091.293398] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5091.653722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5098.533229] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5098.834261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5106.153961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5106.154757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.165782] LustreError: Skipped 1 previous similar message [ 5106.169197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5106.215587] Lustre: Skipped 1 previous similar message [ 5107.671947] Lustre: server umount lustre-OST0000 complete [ 5116.388345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5116.391256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.404176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5116.447470] Lustre: Skipped 1 previous similar message [ 5116.454776] Lustre: Skipped 1 previous similar message [ 5118.868118] Lustre: server umount lustre-MDT0000 complete [ 5123.955557] LustreError: 122784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777498994 with bad export cookie 6150169014161218977 [ 5123.975433] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5124.468853] Lustre: server umount lustre-MDT0001 complete [ 5134.253362] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 17:43:23 (1777499003) [ 5142.037344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.646315] LustreError: 126807:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5142.663365] LustreError: 126807:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 5142.731167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5147.990412] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5148.137310] LustreError: 126808:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5153.250987] LustreError: 126807:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5158.381250] LustreError: 126808:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5158.922869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5159.328772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5165.915454] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5173.674519] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5178.964817] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5183.585828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5183.610371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5183.621666] Lustre: Skipped 2 previous similar messages [ 5188.170248] Lustre: server umount lustre-MDT0000 complete [ 5189.099297] LustreError: 126803:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5189.127297] LustreError: 126803:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 5192.892252] LustreError: 126788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499063 with bad export cookie 6150169014161219859 [ 5192.898715] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5192.906628] LustreError: 126788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5193.251128] Lustre: server umount lustre-MDT0001 complete [ 5204.415803] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 17:44:33 (1777499073) [ 5214.086383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5229.728648] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5234.848295] LustreError: 129001:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 5260.639512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5268.888676] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5276.188210] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5278.918402] Lustre: server umount lustre-OST0000 complete [ 5288.950148] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 17:45:58 (1777499158) [ 5296.147089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5296.811618] LustreError: 130333:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5296.890610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.998818] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5313.086196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5319.400767] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5326.280931] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5330.447619] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5338.014716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5338.383804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.395294] Lustre: Skipped 1 previous similar message [ 5345.034874] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5352.305917] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5363.177073] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5363.188075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5363.207639] Lustre: Skipped 3 previous similar messages [ 5363.229977] Lustre: Skipped 2 previous similar messages [ 5368.306917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5368.315859] Lustre: Skipped 2 previous similar messages [ 5369.627207] Lustre: server umount lustre-MDT0000 complete [ 5373.418746] LustreError: 130334:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.456623] LustreError: 130334:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 5374.933504] LustreError: 130314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499245 with bad export cookie 6150169014161220524 [ 5374.948516] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5374.962883] LustreError: 130314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5375.588644] Lustre: server umount lustre-MDT0001 complete [ 5391.956424] Lustre: server umount lustre-OST0000 complete [ 5402.903519] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 17:47:51 (1777499271) [ 5411.210223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.967993] LustreError: 133649:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5412.131925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5418.051584] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5428.155081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.423272] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5441.930090] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5446.697564] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5455.502323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5457.159068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 5464.891484] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5472.449500] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5478.821599] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5479.213817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5485.585864] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5485.958375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5489.132655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5489.137384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5489.199044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5489.247177] Lustre: Skipped 1 previous similar message [ 5494.743822] Lustre: server umount lustre-OST0000 complete [ 5500.384835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5500.397071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.420063] Lustre: Skipped 1 previous similar message [ 5506.130484] Lustre: server umount lustre-MDT0000 complete [ 5509.603363] LustreError: 134559:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5509.628042] LustreError: 134559:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 5511.787372] LustreError: 136389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499382 with bad export cookie 6150169014161221595 [ 5511.801360] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5511.805386] LustreError: 136389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5512.211033] Lustre: server umount lustre-MDT0001 complete [ 5523.525634] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 17:49:52 (1777499392) [ 5532.541508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5548.258966] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5553.375726] LustreError: 137437:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 5579.108902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5579.117885] Lustre: Skipped 2 previous similar messages [ 5587.330915] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5595.456597] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5603.089747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.730067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5609.550483] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5620.512211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5626.943382] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5634.226482] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5639.803640] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5646.360413] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5646.685662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5652.817884] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5653.044359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5657.058792] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.063132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5657.064408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5657.081070] Lustre: Skipped 3 previous similar messages [ 5657.102678] Lustre: Skipped 7 previous similar messages [ 5661.947132] Lustre: server umount lustre-MDT0000 complete [ 5662.182631] LustreError: 138362:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5662.208326] LustreError: 138362:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 7 previous similar messages [ 5666.216524] LustreError: 137445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499536 with bad export cookie 6150169014161222617 [ 5666.224599] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5666.231243] LustreError: 137445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5666.585390] Lustre: server umount lustre-MDT0001 complete [ 5682.222146] Lustre: server umount lustre-OST0000 complete [ 5693.111233] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 17:52:42 (1777499562) [ 5702.762042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5718.623905] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5723.745619] LustreError: 141262:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 5759.594758] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5767.650878] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5777.092951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.028603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5783.508808] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5794.506264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5800.970376] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5809.346706] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5814.890310] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5824.882993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5825.138536] Lustre: lustre-OST0001: new disk, initializing [ 5825.145291] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5830.748226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5830.757074] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5830.943118] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5834.522635] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5844.947740] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5851.368996] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5851.737164] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5858.092923] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5858.396929] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5861.351977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5861.365289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5861.388754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5867.736538] Lustre: server umount lustre-OST0000 complete [ 5879.005525] Lustre: server umount lustre-OST0001 complete [ 5886.944692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5886.965198] LustreError: Skipped 2 previous similar messages [ 5898.207699] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5898.406172] Lustre: server umount lustre-MDT0000 complete [ 5902.612703] LustreError: 141269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499773 with bad export cookie 6150169014161223569 [ 5902.617535] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5902.631870] LustreError: 141269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5903.118427] Lustre: server umount lustre-MDT0001 complete [ 5933.390339] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 17:56:42 (1777499802) [ 5935.901766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5937.912071] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5940.089564] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 17:56:49 (1777499809) [ 5948.195254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5948.701455] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5948.735383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5949.089545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5949.094683] Lustre: Skipped 6 previous similar messages [ 5954.407796] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5965.754173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.924908] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5965.993907] Lustre: 147215:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5972.239588] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 5980.238898] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5985.190548] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5993.730594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5993.927900] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6002.353410] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6002.964266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 6009.242550] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6015.369568] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6015.692451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6022.131207] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6022.450522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6028.773770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6028.774723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.792295] LustreError: Skipped 1 previous similar message [ 6028.792963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6028.792967] Lustre: Skipped 12 previous similar messages [ 6028.838782] Lustre: Skipped 6 previous similar messages [ 6031.268159] Lustre: server umount lustre-OST0000 complete [ 6056.708354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6066.081293] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6068.530213] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.548335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.549575] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6068.588119] Lustre: Skipped 1 previous similar message [ 6072.748882] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6078.355475] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6078.635197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6083.985693] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6084.413772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6089.797338] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a7c98400800.ost_server_uuid 50 [ 6091.622278] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9a7c98400800.ost_server_uuid in FULL state after 0 sec [ 6091.931254] LustreError: 148162:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.19@tcp) returned error from blocking AST (req@0000000001bf7c7c x1863842604009472 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000040182b04/0x5559ca2225b1b8eb lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.19@tcp remote: 0x621cb25b97ef3f51 expref: 5 pid: 147223 timeout: 6191 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6091.988619] LustreError: lustre-MDT0000: A client on nid 192.168.201.19@tcp was evicted due to a lock blocking callback time out: rc -107 [ 6092.003981] LustreError: 147211:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.201.19@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000040182b04/0x5559ca2225b1b8eb lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.19@tcp remote: 0x621cb25b97ef3f51 expref: 6 pid: 147223 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6099.431727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6104.281898] Lustre: server umount lustre-OST0000 complete [ 6115.060711] Lustre: server umount lustre-MDT0000 complete [ 6119.904654] LustreError: 147226:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6119.938674] LustreError: 147226:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 15 previous similar messages [ 6120.040241] LustreError: 147207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777499990 with bad export cookie 6150169014161224752 [ 6120.049398] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6120.059604] LustreError: 147207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6126.620200] Lustre: server umount lustre-MDT0001 complete [ 6139.301576] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 6142.560110] Key type lgssc unregistered [ 6142.940584] LNet: 152700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6142.971159] LNetError: 152700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6143.023157] LNet: Removed LNI 192.168.201.119@tcp [ 6143.946626] Key type .llcrypt unregistered [ 6143.954744] Key type ._llcrypt unregistered [ 6170.193720] Key type ._llcrypt registered [ 6170.196643] Key type .llcrypt registered [ 6170.311731] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 6171.756822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6172.062738] alg: No test for adler32 (adler32-zlib) [ 6173.222968] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6173.465534] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 6175.263940] Key type lgssc registered [ 6176.659709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6185.144403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6185.183925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.805353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6192.370680] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6197.223608] LustreError: 154098:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6197.253146] LustreError: 154098:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6202.067518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6202.345943] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6202.646062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6208.315961] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6214.514691] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6218.631242] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6226.483858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6226.859322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.048360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6233.353421] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6239.743569] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6247.716503] Lustre: Failing over lustre-MDT0000 [ 6247.888769] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 6248.227806] Lustre: server umount lustre-MDT0000 complete [ 6248.931402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6248.938803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.959595] LustreError: 154098:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6252.516152] LustreError: 154097:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6252.522053] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6257.635096] LustreError: 155029:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.682168] LustreError: 155029:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 6261.114918] LustreError: 156349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.19@tcp arrived at 1777500131 with bad export cookie 10170215983360482268 [ 6261.147937] LustreError: 156349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6262.789930] LustreError: 155203:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.846805] LustreError: 155203:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6268.895305] Lustre: 153407:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777500123/real 1777500123] req@00000000b8034206 x1863843886164736/t0(0) o400->MGC192.168.201.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777500139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6268.980111] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6270.820783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6274.016876] LustreError: 155029:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6274.032580] LustreError: 155029:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 6279.144365] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8d23dd8d81339c3e [ 6279.158888] Lustre: MGC192.168.201.119@tcp: Connection restored to 0@lo (at 0@lo) [ 6279.431549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6279.490136] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6284.004159] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6284.779583] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6284.790918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6286.380273] Lustre: lustre-MDT0000: Denying connection for new client c20662ed-fa74-45cd-864b-eafd4344d8f3 (at 192.168.201.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 6295.010839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.038054] Lustre: Skipped 1 previous similar message [ 6295.045791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6295.058809] Lustre: Skipped 1 previous similar message [ 6300.146355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.156839] Lustre: Skipped 1 previous similar message [ 6305.257392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6305.267841] Lustre: Skipped 1 previous similar message [ 6308.832789] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6309.056036] Lustre: server umount lustre-OST0000 complete [ 6313.001471] LustreError: 157486:0:(ldlm_lib.c:2986:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 6313.010896] Lustre: 156850:0:(ldlm_lib.c:2389:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6313.018889] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 6313.042350] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 6313.053990] LustreError: 156850:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a4355e79 x1863843886186112/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6313.070946] LustreError: 156850:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 6313.078922] LustreError: 156850:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 6313.088352] Lustre: lustre-MDT0000: Recovery over after 0:29, of 2 clients 0 recovered and 2 were evicted. [ 6313.373970] Lustre: server umount lustre-MDT0000 complete [ 6315.490615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6315.519730] Lustre: Skipped 1 previous similar message [ 6315.525906] LustreError: 154092:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6315.556155] LustreError: 154092:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 6317.381793] LustreError: 154076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500188 with bad export cookie 10170215983360482366 [ 6317.386588] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6317.395131] LustreError: 154076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6317.395557] LustreError: 153406:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 6319.803685] Lustre: server umount lustre-MDT0001 complete [ 6332.648733] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 6335.835752] Key type lgssc unregistered [ 6336.160392] LNet: 158223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6336.179733] LNetError: 158223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6337.273266] LNet: Removed LNI 192.168.201.119@tcp [ 6338.300342] Key type .llcrypt unregistered [ 6338.303682] Key type ._llcrypt unregistered [ 6362.915395] Key type ._llcrypt registered [ 6362.920267] Key type .llcrypt registered [ 6363.072082] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 6363.965964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6363.983257] alg: No test for adler32 (adler32-zlib) [ 6365.218718] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6365.533727] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 6367.303182] Key type lgssc registered [ 6368.902347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6378.357029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6378.378280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6380.105034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6385.338571] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6390.253934] LustreError: 159620:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6390.280862] LustreError: 159620:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6393.816510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.240116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6398.886863] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6405.040504] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6409.029740] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6415.379816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6415.886870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6422.521254] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6425.144577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6428.108647] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6435.299617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6435.304119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.320605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6435.329958] Lustre: Skipped 1 previous similar message [ 6435.341321] Lustre: Skipped 1 previous similar message [ 6440.437588] LustreError: 161214:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6440.479090] LustreError: 161214:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6440.628347] Lustre: server umount lustre-OST0000 complete [ 6445.537895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.543401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6445.550904] Lustre: Skipped 1 previous similar message [ 6445.551262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6445.556831] Lustre: Skipped 1 previous similar message [ 6450.572380] Lustre: server umount lustre-MDT0000 complete [ 6450.674625] LustreError: 161395:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6450.701755] LustreError: 161395:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6454.154108] LustreError: 159600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500324 with bad export cookie 17314676453699530601 [ 6454.163180] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6454.167897] LustreError: 159600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6454.474269] Lustre: server umount lustre-MDT0001 complete [ 6464.402217] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 6467.042194] Key type lgssc unregistered [ 6467.347824] LNet: 162797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6467.368362] LNetError: 162797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6467.388219] LNet: Removed LNI 192.168.201.119@tcp [ 6468.206807] Key type .llcrypt unregistered [ 6468.208572] Key type ._llcrypt unregistered [ 6500.453902] Key type ._llcrypt registered [ 6500.458094] Key type .llcrypt registered [ 6500.590469] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 6501.624240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6501.727384] alg: No test for adler32 (adler32-zlib) [ 6502.832967] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6503.082245] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 6504.800849] Key type lgssc registered [ 6505.888635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6513.744107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6513.764962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.177089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6519.837978] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6524.389141] LustreError: 164483:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6524.418891] LustreError: 164483:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6528.498213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6528.998410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6532.506096] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6538.213979] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6542.501409] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6549.646133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6550.134750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6556.571923] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6560.371978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6562.285572] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6569.953419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6569.969353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6569.999470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6570.991850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6570.994566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6570.997695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.767245] Lustre: server umount lustre-OST0000 complete [ 6580.202809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6580.221731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6580.244763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6580.249742] Lustre: Skipped 1 previous similar message [ 6584.644753] Lustre: server umount lustre-MDT0000 complete [ 6586.337932] LustreError: 165846:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6586.354866] LustreError: 165846:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6588.392971] LustreError: 166923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500459 with bad export cookie 2695702205165553419 [ 6588.402577] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6588.408837] LustreError: 166923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6588.695908] Lustre: server umount lustre-MDT0001 complete [ 6598.421876] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 6600.955665] Key type lgssc unregistered [ 6601.205338] LNet: 167657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6601.210036] LNetError: 167657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6601.229897] LNet: Removed LNI 192.168.201.119@tcp [ 6601.947864] Key type .llcrypt unregistered [ 6601.950682] Key type ._llcrypt unregistered [ 6621.744417] Key type ._llcrypt registered [ 6621.747645] Key type .llcrypt registered [ 6621.871482] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 6623.250843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6623.272787] alg: No test for adler32 (adler32-zlib) [ 6624.410842] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6624.650859] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 6626.295470] Key type lgssc registered [ 6627.493227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6638.863558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6638.877343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.288403] Lustre: *** cfs_fail_loc=135, val=0*** [ 6640.411872] LustreError: 169145:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6640.419938] LustreError: 169145:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.119@tcp: cfg command failed: rc = -2 [ 6640.432519] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6640.432519] [ 6640.447856] LustreError: MGC192.168.201.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6640.469129] LustreError: 169106:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6640.479665] LustreError: 169106:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 6640.488614] LustreError: 169106:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 6640.499820] LustreError: 169106:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6640.605165] Lustre: server umount lustre-MDT0000 complete [ 6640.612426] LustreError: 169106:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6647.414086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.763229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.259446] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6658.022349] LustreError: 169516:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6658.056921] LustreError: 169516:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6659.419201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.847741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6664.402855] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6669.788937] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6673.175509] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6675.443076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6675.453706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.477610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6678.497562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6678.497959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.438979] Lustre: server umount lustre-MDT0000 complete [ 6683.616846] LustreError: 169513:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.629866] LustreError: 169513:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6684.644746] LustreError: 169510:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6684.662043] LustreError: 169510:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6684.919704] LustreError: 169497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500555 with bad export cookie 447403019062400840 [ 6684.933514] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6684.942875] LustreError: 169497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6685.190225] Lustre: server umount lustre-MDT0001 complete [ 6696.525143] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 6699.361081] Key type lgssc unregistered [ 6699.743525] LNet: 171596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6699.755879] LNetError: 171596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6699.781589] LNet: Removed LNI 192.168.201.119@tcp [ 6700.717849] Key type .llcrypt unregistered [ 6700.721363] Key type ._llcrypt unregistered [ 6722.157495] Key type ._llcrypt registered [ 6722.158594] Key type .llcrypt registered [ 6722.255468] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 6722.540517] Key type .llcrypt unregistered [ 6722.545631] Key type ._llcrypt unregistered [ 6739.076337] Key type ._llcrypt registered [ 6739.096519] Key type .llcrypt registered [ 6739.193771] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 6740.524106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6740.543417] alg: No test for adler32 (adler32-zlib) [ 6741.575623] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6741.827253] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 6743.463174] Key type lgssc registered [ 6744.626805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6752.965671] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6752.983639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6768.543237] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6773.663399] LustreError: 173697:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.119@tcp: failed processing log, type 4: rc = -110 [ 6800.415126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6806.905858] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6812.482395] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6819.544866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6820.041821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6820.106960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6823.772551] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6825.451304] LustreError: 173722:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6825.485772] LustreError: 173722:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 6830.563693] LustreError: 173723:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6832.287989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.646876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6836.994896] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6842.234814] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6845.582918] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6848.037560] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6860.261923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6860.273880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6860.274363] Lustre: Skipped 1 previous similar message [ 6860.285898] Lustre: Skipped 1 previous similar message [ 6865.378901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6865.402745] Lustre: Skipped 1 previous similar message [ 6865.791433] Lustre: server umount lustre-OST0000 complete [ 6870.509037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6870.524490] Lustre: Skipped 1 previous similar message [ 6870.526707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6870.551579] Lustre: Skipped 1 previous similar message [ 6875.622389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6875.628116] Lustre: Skipped 1 previous similar message [ 6880.737563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6880.748488] Lustre: Skipped 1 previous similar message [ 6883.300381] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6883.463416] Lustre: server umount lustre-MDT0000 complete [ 6885.860629] LustreError: 174625:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.868199] LustreError: 174625:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 6886.912213] LustreError: 175343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500757 with bad export cookie 1773700888173931942 [ 6886.914778] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6886.923427] LustreError: 175343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6887.280711] Lustre: server umount lustre-MDT0001 complete [ 6896.933578] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 6899.558753] Key type lgssc unregistered [ 6899.797440] LNet: 177353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6899.803624] LNetError: 177353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6899.822434] LNet: Removed LNI 192.168.201.119@tcp [ 6900.520236] Key type .llcrypt unregistered [ 6900.523746] Key type ._llcrypt unregistered [ 6921.828071] Key type ._llcrypt registered [ 6921.832333] Key type .llcrypt registered [ 6921.931979] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 6922.828272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6922.854500] alg: No test for adler32 (adler32-zlib) [ 6923.864981] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 6924.002937] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 6925.647265] Key type lgssc registered [ 6926.675324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6932.482369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6932.499631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6933.863015] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6937.422534] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6944.237117] LustreError: 178750:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6944.257397] LustreError: 178750:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 6944.725470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6945.119844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6948.660109] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6953.162957] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6956.274129] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6962.253413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6962.615639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6968.212838] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 6972.991437] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6979.005820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6982.482441] Lustre: Failing over lustre-MDT0000 [ 6982.780653] Lustre: server umount lustre-MDT0000 complete [ 6984.165084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.170141] LustreError: 178751:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.182643] Lustre: Skipped 1 previous similar message [ 6989.289824] LustreError: 178775:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.315140] LustreError: 178775:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 6994.407683] LustreError: 179463:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.438786] LustreError: 179463:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 6999.398108] Lustre: 178060:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777500854/real 1777500854] req@00000000ffe5e330 x1863844672593152/t0(0) o400->MGC192.168.201.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1777500870 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6999.451523] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6999.472789] LustreError: 178751:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.504689] LustreError: 178751:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7000.474141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.763628] LustreError: 178747:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7009.789891] LustreError: 178747:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 7010.106522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7010.167487] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7010.203961] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7014.747794] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7015.406091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7015.417133] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7015.468599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 7019.001633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 7025.786506] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 7035.467836] Lustre: server umount lustre-OST0000 complete [ 7040.992607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7040.994753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.995097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7041.049294] Lustre: Skipped 2 previous similar messages [ 7046.114921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7046.120521] Lustre: Skipped 2 previous similar messages [ 7049.299920] Lustre: server umount lustre-MDT0000 complete [ 7051.243939] LustreError: 178746:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.272891] LustreError: 178746:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7052.259295] Lustre: 178061:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1777500906/real 1777500906] req@000000003e2c9d45 x1863844672623616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1777500922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7052.278194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.636308] LustreError: 178732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777500923 with bad export cookie 1181760617203880393 [ 7052.640624] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7052.653256] LustreError: 178732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7052.967956] Lustre: server umount lustre-MDT0001 complete [ 7062.110573] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 7064.330526] Key type lgssc unregistered [ 7064.574121] LNet: 183262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7064.578630] LNetError: 183262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7064.595763] LNet: Removed LNI 192.168.201.119@tcp [ 7065.275329] Key type .llcrypt unregistered [ 7065.277229] Key type ._llcrypt unregistered [ 7083.378137] Key type ._llcrypt registered [ 7083.380855] Key type .llcrypt registered [ 7083.509491] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7084.437161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7084.594537] alg: No test for adler32 (adler32-zlib) [ 7085.598140] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7085.816917] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 7087.495859] Key type lgssc registered [ 7088.440933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7095.365181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7095.385347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7096.807335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7100.728402] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7107.049211] LustreError: 184637:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7107.070311] LustreError: 184637:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7108.082738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7108.356260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7112.180436] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7117.428804] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7121.599357] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7128.176946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7128.467202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7134.763310] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7139.298576] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7141.825057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 7147.030683] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7149.132213] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 7161.424361] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7172.580194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.580973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7172.608039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.119794] Lustre: server umount lustre-OST0000 complete [ 7180.256523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7180.263349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7180.281685] Lustre: Skipped 1 previous similar message [ 7180.289697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7180.296398] Lustre: Skipped 1 previous similar message [ 7182.817035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7182.821313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.503139] Lustre: server umount lustre-MDT0000 complete [ 7187.939921] LustreError: 184637:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.573294] LustreError: 186255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501059 with bad export cookie 3455690034828058001 [ 7188.577423] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7188.581795] LustreError: 186255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7188.867914] Lustre: server umount lustre-MDT0001 complete [ 7197.112176] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 7199.118046] Key type lgssc unregistered [ 7199.342963] LNet: 187998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7199.349416] LNetError: 187998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7199.363089] LNet: Removed LNI 192.168.201.119@tcp [ 7200.118995] Key type .llcrypt unregistered [ 7200.123144] Key type ._llcrypt unregistered [ 7218.863935] Key type ._llcrypt registered [ 7218.871398] Key type .llcrypt registered [ 7218.968648] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7219.577496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7219.650981] alg: No test for adler32 (adler32-zlib) [ 7220.586503] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7220.720152] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 7222.335821] Key type lgssc registered [ 7223.380557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7229.828873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7229.848213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.294235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7234.763280] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7240.675385] LustreError: 189374:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7240.705379] LustreError: 189374:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7241.947299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7242.247943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7245.058391] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7248.867240] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7251.842857] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7257.235931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7257.525292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7262.636872] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7267.392282] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7275.962088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 7282.394507] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 7296.002958] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7305.375302] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 7312.155727] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 7317.990115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7318.001096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7318.025838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7322.091370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7322.092833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7322.102425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7322.117572] Lustre: Skipped 1 previous similar message [ 7327.207212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7327.216540] Lustre: Skipped 1 previous similar message [ 7331.295126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7331.527346] Lustre: server umount lustre-OST0000 complete [ 7334.369251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7334.372951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.377995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7337.443572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7340.433947] Lustre: server umount lustre-MDT0000 complete [ 7342.560129] LustreError: 190085:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7342.572552] LustreError: 190085:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7342.865332] LustreError: 189354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501213 with bad export cookie 10911544033488807183 [ 7342.870423] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7342.878241] LustreError: 189354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7343.086281] Lustre: server umount lustre-MDT0001 complete [ 7349.646181] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 7351.748447] Key type lgssc unregistered [ 7352.011198] LNet: 194056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7352.022749] LNetError: 194056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7352.035654] LNet: Removed LNI 192.168.201.119@tcp [ 7352.721610] Key type .llcrypt unregistered [ 7352.723967] Key type ._llcrypt unregistered [ 7369.458208] Key type ._llcrypt registered [ 7369.460488] Key type .llcrypt registered [ 7369.529233] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7370.523962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7370.545765] alg: No test for adler32 (adler32-zlib) [ 7371.575648] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7371.792542] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 7373.471561] Key type lgssc registered [ 7374.465236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7380.452959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7380.466705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7381.839436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7384.955770] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7391.519737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7391.762563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7394.810086] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7399.368727] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7402.279791] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7407.506910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7407.888899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7408.937566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7413.036690] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7417.217900] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7429.600770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7429.607486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7429.614514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7432.675408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7432.681347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7432.706846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7434.238729] Lustre: server umount lustre-OST0000 complete [ 7437.795149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7437.805418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7437.806791] Lustre: Skipped 1 previous similar message [ 7437.806865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7437.806869] Lustre: Skipped 1 previous similar message [ 7442.912977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7442.922892] Lustre: Skipped 2 previous similar messages [ 7443.232322] Lustre: server umount lustre-MDT0000 complete [ 7445.907266] LustreError: 195415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501316 with bad export cookie 1230267810796807648 [ 7445.913119] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7445.916250] LustreError: 195415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7446.118190] Lustre: server umount lustre-MDT0001 complete [ 7454.055504] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 7456.168679] Key type lgssc unregistered [ 7456.376874] LNet: 198757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7456.379827] LNetError: 198757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7456.387741] LNet: Removed LNI 192.168.201.119@tcp [ 7456.913115] Key type .llcrypt unregistered [ 7456.914736] Key type ._llcrypt unregistered [ 7468.181411] Key type ._llcrypt registered [ 7468.183072] Key type .llcrypt registered [ 7468.261254] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7468.795367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7468.874205] alg: No test for adler32 (adler32-zlib) [ 7469.865542] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7470.083193] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 7471.719533] Key type lgssc registered [ 7472.487823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7478.186448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7478.203530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7479.552452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.937464] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7489.662307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7489.978829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7493.484294] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7497.471978] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7500.151087] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7505.047755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7505.347632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7506.421440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7509.643316] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7512.977107] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7524.837742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7524.839749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7524.870517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7524.894814] Lustre: Skipped 1 previous similar message [ 7529.117994] Lustre: server umount lustre-OST0000 complete [ 7535.079528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7535.090593] Lustre: Skipped 1 previous similar message [ 7535.096340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7535.105916] Lustre: Skipped 1 previous similar message [ 7538.318467] Lustre: server umount lustre-MDT0000 complete [ 7540.195969] LustreError: 199984:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7540.208107] LustreError: 199984:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7540.894497] LustreError: 199965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501411 with bad export cookie 11811375391967971269 [ 7540.898424] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7540.906692] LustreError: 199965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7541.149354] Lustre: server umount lustre-MDT0001 complete [ 7548.411291] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 7550.225378] Key type lgssc unregistered [ 7550.390329] LNet: 203309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7550.393180] LNetError: 203309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7550.402745] LNet: Removed LNI 192.168.201.119@tcp [ 7550.903056] Key type .llcrypt unregistered [ 7550.904586] Key type ._llcrypt unregistered [ 7564.327703] Key type ._llcrypt registered [ 7564.331703] Key type .llcrypt registered [ 7564.443781] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7565.276103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7565.385407] alg: No test for adler32 (adler32-zlib) [ 7566.495121] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7566.725774] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 7568.391222] Key type lgssc registered [ 7569.506217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7575.760774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7575.774268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7577.132529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7579.890964] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7585.890205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7586.225676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7590.005432] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7594.079757] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7596.830299] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7601.495623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7601.771673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7605.313768] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7608.287597] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7622.252572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7629.625658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7629.691050] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7629.836870] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7633.620970] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7636.949917] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7640.244947] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 7651.500565] Lustre: Permanently deactivating lustre-OST0001 [ 7651.526956] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7660.554208] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7669.169843] Lustre: Permanently reactivating lustre-OST0001 [ 7669.186236] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7669.188985] Lustre: Skipped 2 previous similar messages [ 7669.234506] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7669.239146] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7669.242200] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7669.245479] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7669.268806] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7683.558210] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.558939] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7683.565807] Lustre: Skipped 2 previous similar messages [ 7685.844777] Lustre: server umount lustre-OST0001 complete [ 7688.676376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7688.677882] LustreError: 206131:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.677977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7688.677985] Lustre: Skipped 1 previous similar message [ 7688.684342] Lustre: Skipped 1 previous similar message [ 7688.705302] LustreError: 206131:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 7693.794566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7693.798188] Lustre: Skipped 1 previous similar message [ 7693.803628] LustreError: 206131:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7693.812884] LustreError: 206131:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7694.674453] Lustre: server umount lustre-OST0000 complete [ 7698.913744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7698.914530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.919678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7698.929267] Lustre: Skipped 1 previous similar message [ 7703.205699] Lustre: server umount lustre-MDT0000 complete [ 7704.032240] LustreError: 204536:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7704.048962] LustreError: 204536:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 7705.905080] LustreError: 204517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501576 with bad export cookie 12522677793681053514 [ 7705.909365] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7705.909789] LustreError: 204517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7706.095700] Lustre: server umount lustre-MDT0001 complete [ 7712.920887] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 7714.667071] Key type lgssc unregistered [ 7714.859376] LNet: 209185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7714.867793] LNetError: 209185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7714.878347] LNet: Removed LNI 192.168.201.119@tcp [ 7715.443504] Key type .llcrypt unregistered [ 7715.446200] Key type ._llcrypt unregistered [ 7736.531185] Key type ._llcrypt registered [ 7736.533315] Key type .llcrypt registered [ 7736.594346] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7737.175637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7737.188816] alg: No test for adler32 (adler32-zlib) [ 7738.113240] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7738.230906] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 7739.839174] Key type lgssc registered [ 7740.319223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7743.993127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7744.003073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.224063] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7745.241670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7745.351296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7747.617339] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7752.175275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7752.278940] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7752.304352] Lustre: 211349:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7752.483563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7754.862313] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7758.069350] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7760.273424] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7764.510285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7764.631350] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7764.760365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.056988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7767.946299] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7770.720571] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7772.580553] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7777.471207] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7784.430763] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7792.296472] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7802.415506] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7822.257805] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7822.263564] Lustre: Skipped 2 previous similar messages [ 7856.150975] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7856.159623] Lustre: Skipped 3 previous similar messages [ 7923.298500] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7923.302258] Lustre: Skipped 7 previous similar messages [ 7944.160994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7944.162603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7944.171567] LustreError: Skipped 1 previous similar message [ 7944.178898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7946.546646] Lustre: server umount lustre-OST0000 complete [ 7949.282349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7949.294312] Lustre: Skipped 1 previous similar message [ 7949.299064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7949.301699] Lustre: Skipped 2 previous similar messages [ 7954.407888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7954.415506] Lustre: Skipped 1 previous similar message [ 7954.911940] Lustre: server umount lustre-MDT0000 complete [ 7956.900101] LustreError: 212071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501827 with bad export cookie 13386926155429380500 [ 7956.903527] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7956.905658] LustreError: 212071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7957.077133] Lustre: server umount lustre-MDT0001 complete [ 7962.838660] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 7964.711650] Key type lgssc unregistered [ 7964.947705] LNet: 215801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7964.951607] LNetError: 215801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7964.960724] LNet: Removed LNI 192.168.201.119@tcp [ 7965.472420] Key type .llcrypt unregistered [ 7965.474228] Key type ._llcrypt unregistered [ 7978.637031] Key type ._llcrypt registered [ 7978.639011] Key type .llcrypt registered [ 7978.697341] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 7979.224815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7979.286464] alg: No test for adler32 (adler32-zlib) [ 7980.185923] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 7980.322968] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 7981.959256] Key type lgssc registered [ 7982.728600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7987.447513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7987.463200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.752744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.072483] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 7993.824354] LustreError: 217176:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7993.832580] LustreError: 217176:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 7996.417530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7996.684320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7999.065163] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8002.017199] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8004.060561] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8007.966508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8008.219521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8012.364504] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug -1 all [ 8014.378931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 8015.766864] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8029.665360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8029.672936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.681517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.756944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.764601] Lustre: Skipped 1 previous similar message [ 8034.429969] Lustre: server umount lustre-OST0000 complete [ 8036.824558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8036.836746] Lustre: Skipped 1 previous similar message [ 8036.843128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8036.846365] Lustre: Skipped 1 previous similar message [ 8037.860425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8037.869878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.953044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8041.958924] Lustre: Skipped 2 previous similar messages [ 8047.077187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8047.082791] Lustre: Skipped 1 previous similar message [ 8051.167150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8051.282640] Lustre: server umount lustre-MDT0000 complete [ 8052.194918] LustreError: 217172:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8052.206795] LustreError: 217172:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 8053.716949] LustreError: 217156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777501924 with bad export cookie 10589220346040896248 [ 8053.717759] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8053.723144] LustreError: 217156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8053.939395] Lustre: server umount lustre-MDT0001 complete [ 8060.690895] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 8062.626489] Key type lgssc unregistered [ 8062.819851] LNet: 220510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8062.828457] LNetError: 220510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8062.845614] LNet: Removed LNI 192.168.201.119@tcp [ 8063.361258] Key type .llcrypt unregistered [ 8063.363437] Key type ._llcrypt unregistered [ 8077.913269] Key type ._llcrypt registered [ 8077.915766] Key type .llcrypt registered [ 8077.996487] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 8078.506456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8078.552544] alg: No test for adler32 (adler32-zlib) [ 8079.589092] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 8079.779156] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 8081.407728] Key type lgssc registered [ 8082.182315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8348.223236] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 8350.180169] Key type lgssc unregistered [ 8350.393151] LNet: 222472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8350.401746] LNetError: 222472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8350.413447] LNet: Removed LNI 192.168.201.119@tcp [ 8350.906255] Key type .llcrypt unregistered [ 8350.908626] Key type ._llcrypt unregistered [ 8367.531452] Key type ._llcrypt registered [ 8367.533301] Key type .llcrypt registered [ 8367.611672] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 8378.283253] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 8378.858559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8378.932908] alg: No test for adler32 (adler32-zlib) [ 8379.908211] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 8380.051699] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 8381.672238] Key type lgssc registered [ 8382.461571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8400.112400] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8043 sec ========== 18:37:49 (1777502269) [ 8401.288737] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:37:51 (1777502271) === [ 8403.728735] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:37:53 (1777502273) === [ 8428.580424] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 8430.417573] Key type lgssc unregistered [ 8430.626464] LNet: 227632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8430.634272] LNetError: 227632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8430.653145] LNet: Removed LNI 192.168.201.119@tcp [ 8431.185529] Key type .llcrypt unregistered [ 8431.188221] Key type ._llcrypt unregistered