[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 524729345 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002395] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007553] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010121] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.013019] SELinux: Initializing. [ 0.014073] *** VALIDATE selinux *** [ 0.022636] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026797] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028011] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031107] *** VALIDATE tmpfs *** [ 0.032464] *** VALIDATE proc *** [ 0.033262] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035255] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.041782] debug: unmapping init [mem 0xffffffffa7c59000-0xffffffffa7c60fff] [ 0.043788] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044734] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045023] ... version: 2 [ 0.046016] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048017] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050018] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052310] rcu: Hierarchical SRCU implementation. [ 0.054364] smp: Bringing up secondary CPUs ... [ 0.055481] x86: Booting SMP configuration: [ 0.056025] .... node #0, CPUs: #1 #2 #3 [ 0.059321] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147555] node 0 deferred pages initialised in 83ms [ 0.151191] devtmpfs: initialized [ 0.153275] x86/mm: Memory block size: 128MB [ 0.156914] gcov: version magic: 0x41383552 [ 0.160352] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166336] pinctrl core: initialized pinctrl subsystem [ 0.169162] [ 0.169675] ************************************************************* [ 0.172013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174012] ** ** [ 0.177012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.180012] ** ** [ 0.182012] ** This means that this kernel is built to expose internal ** [ 0.185013] ** IOMMU data structures, which may compromise security on ** [ 0.188013] ** your system. ** [ 0.191013] ** ** [ 0.193012] ** If you see this message and you are not debugging the ** [ 0.196012] ** kernel, report this immediately to your vendor! ** [ 0.198013] ** ** [ 0.201010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202007] ************************************************************* [ 0.205033] NET: Registered protocol family 16 [ 0.207428] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.210063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.213059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218033] cpuidle: using governor menu [ 0.219765] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.223609] PCI: Using configuration type 1 for base access [ 0.226124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.235079] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.237020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.241035] cryptd: max_cpu_qlen set to 1000 [ 0.244253] ACPI: Added _OSI(Module Device) [ 0.246026] ACPI: Added _OSI(Processor Device) [ 0.248013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.255163] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.261303] ACPI: Interpreter enabled [ 0.263060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.265013] ACPI: Using IOAPIC for interrupt routing [ 0.267111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.271364] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.281821] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.285041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.288017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.291091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.298250] acpiphp: Slot [2] registered [ 0.299134] acpiphp: Slot [5] registered [ 0.300148] acpiphp: Slot [6] registered [ 0.302144] acpiphp: Slot [7] registered [ 0.304323] acpiphp: Slot [8] registered [ 0.306178] acpiphp: Slot [9] registered [ 0.307305] acpiphp: Slot [10] registered [ 0.310144] acpiphp: Slot [3] registered [ 0.311103] acpiphp: Slot [4] registered [ 0.313128] acpiphp: Slot [11] registered [ 0.314078] acpiphp: Slot [12] registered [ 0.316188] acpiphp: Slot [13] registered [ 0.317129] acpiphp: Slot [14] registered [ 0.319087] acpiphp: Slot [15] registered [ 0.321110] acpiphp: Slot [16] registered [ 0.322153] acpiphp: Slot [17] registered [ 0.324111] acpiphp: Slot [18] registered [ 0.326087] acpiphp: Slot [19] registered [ 0.327127] acpiphp: Slot [20] registered [ 0.329103] acpiphp: Slot [21] registered [ 0.331101] acpiphp: Slot [22] registered [ 0.332085] acpiphp: Slot [23] registered [ 0.334114] acpiphp: Slot [24] registered [ 0.336112] acpiphp: Slot [25] registered [ 0.337123] acpiphp: Slot [26] registered [ 0.339107] acpiphp: Slot [27] registered [ 0.341114] acpiphp: Slot [28] registered [ 0.342103] acpiphp: Slot [29] registered [ 0.344097] acpiphp: Slot [30] registered [ 0.346098] acpiphp: Slot [31] registered [ 0.347058] PCI host bridge to bus 0000:00 [ 0.349020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.351024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.354030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.357027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.360024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.363027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.364228] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.368027] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.371364] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.382000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.387058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.390020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.393020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.397020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.400475] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.403787] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.407111] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.409758] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.415016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.428954] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.435856] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.441658] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.451016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.459015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.475025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.485000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.494015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.502016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.531019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.542697] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.558016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.565023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.581025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.597601] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.609018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.617015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.635016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.651365] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.661020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.668020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.686026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.697850] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.703098] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.710021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.739027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.751768] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.757552] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.760532] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.763553] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.766295] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.772001] iommu: Default domain type: Passthrough [ 0.774464] SCSI subsystem initialized [ 0.776219] ACPI: bus type USB registered [ 0.778116] usbcore: registered new interface driver usbfs [ 0.780099] usbcore: registered new interface driver hub [ 0.782091] usbcore: registered new device driver usb [ 0.785215] pps_core: LinuxPPS API ver. 1 registered [ 0.787012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.791065] PTP clock support registered [ 0.793154] EDAC MC: Ver: 3.0.0 [ 0.795237] PCI: Using ACPI for IRQ routing [ 0.797808] NetLabel: Initializing [ 0.798016] NetLabel: domain hash size = 128 [ 0.800016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.802105] NetLabel: unlabeled traffic allowed by default [ 0.804172] vgaarb: loaded [ 0.806444] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.810025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.817320] clocksource: Switched to clocksource kvm-clock [ 0.924580] VFS: Disk quotas dquot_6.6.0 [ 0.926379] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.929942] *** VALIDATE ramfs *** [ 0.931383] *** VALIDATE hugetlbfs *** [ 0.933219] pnp: PnP ACPI init [ 0.935778] pnp: PnP ACPI: found 6 devices [ 0.951526] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.955149] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.957704] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.960278] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.963126] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.966049] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.969422] NET: Registered protocol family 2 [ 0.972348] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.977541] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.981415] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.986672] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.990351] TCP: Hash tables configured (established 65536 bind 65536) [ 0.993682] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.997358] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.024429] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.028136] NET: Registered protocol family 1 [ 1.031143] RPC: Registered named UNIX socket transport module. [ 1.033792] RPC: Registered udp transport module. [ 1.035741] RPC: Registered tcp transport module. [ 1.036757] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.039685] NET: Registered protocol family 44 [ 1.041132] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.043748] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.046335] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.049843] PCI: CLS 0 bytes, default 64 [ 1.052333] Unpacking initramfs... [ 2.462092] debug: unmapping init [mem 0xffff94e33cc54000-0xffff94e33ffbffff] [ 2.468705] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.471270] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.474359] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.973792] Initialise system trusted keyrings [ 2.977917] Key type blacklist registered [ 2.980683] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.990597] zbud: loaded [ 2.993862] *** VALIDATE nfs *** [ 2.995388] *** VALIDATE nfs4 *** [ 2.997162] pstore: using deflate compression [ 3.000759] Platform Keyring initialized [ 3.110493] NET: Registered protocol family 38 [ 3.112459] Key type asymmetric registered [ 3.114323] Asymmetric key parser 'x509' registered [ 3.116373] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.119828] io scheduler mq-deadline registered [ 3.122436] io scheduler kyber registered [ 3.124337] io scheduler bfq registered [ 3.128484] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.131898] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.135631] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.139088] ACPI: Power Button [PWRF] [ 3.144652] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.152403] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.177640] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.185267] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.203687] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.230520] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.259671] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.264482] Non-volatile memory driver v1.3 [ 3.266054] Linux agpgart interface v0.103 [ 3.299827] virtio_blk virtio1: [vda] 150024 512-byte logical blocks (76.8 MB/73.3 MiB) [ 3.303976] vda: detected capacity change from 0 to 76812288 [ 3.320233] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.324548] vdb: detected capacity change from 0 to 1073741824 [ 3.343276] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.347236] vdc: detected capacity change from 0 to 2621440000 [ 3.363807] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.366780] vdd: detected capacity change from 0 to 2621440000 [ 3.385838] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.389655] vde: detected capacity change from 0 to 4294967296 [ 3.406405] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.409178] vdf: detected capacity change from 0 to 4294967296 [ 3.415415] libphy: Fixed MDIO Bus: probed [ 3.421254] usbcore: registered new interface driver usbserial_generic [ 3.425697] usbserial: USB Serial support registered for generic [ 3.429280] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.434958] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.437591] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.440711] mousedev: PS/2 mouse device common for all mice [ 3.444737] rtc_cmos 00:05: RTC can wake from S4 [ 3.449477] rtc_cmos 00:05: registered as rtc0 [ 3.451903] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.455818] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.460245] intel_pstate: CPU model not supported [ 3.460843] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.467744] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.467797] hid: raw HID events driver (C) Jiri Kosina [ 3.473158] usbcore: registered new interface driver usbhid [ 3.475712] usbhid: USB HID core driver [ 3.477915] drop_monitor: Initializing network drop monitor service [ 3.480236] Initializing XFRM netlink socket [ 3.481887] NET: Registered protocol family 10 [ 3.484767] Segment Routing with IPv6 [ 3.486677] NET: Registered protocol family 17 [ 3.489082] mpls_gso: MPLS GSO support [ 3.496718] RAS: Correctable Errors collector initialized. [ 3.499960] AVX version of gcm_enc/dec engaged. [ 3.501853] AES CTR mode by8 optimization enabled [ 3.584811] sched_clock: Marking stable (3584786861, 0)->(4437157119, -852370258) [ 3.588682] registered taskstats version 1 [ 3.591297] Loading compiled-in X.509 certificates [ 3.594131] zswap: loaded using pool lzo/zbud [ 3.621065] Key type big_key registered [ 3.634083] Key type encrypted registered [ 3.636249] ima: No TPM chip found, activating TPM-bypass! [ 3.638627] ima: Allocated hash algorithm: sha1 [ 3.640431] ima: No architecture policies found [ 3.642430] evm: Initialising EVM extended attributes: [ 3.644640] evm: security.selinux [ 3.645932] evm: security.ima [ 3.647347] evm: security.capability [ 3.648795] evm: HMAC attrs: 0x1 [ 3.651333] rtc_cmos 00:05: setting system clock to 2026-09-14 23:25:26 UTC (1789428326) [ 3.658396] debug: unmapping init [mem 0xffffffffa8c03000-0xffffffffa8dfffff] [ 3.662113] debug: unmapping init [mem 0xffffffffa7982000-0xffffffffa7c58fff] [ 3.670412] Write protecting the kernel read-only data: 28672k [ 3.674371] debug: unmapping init [mem 0xffffffffa6003000-0xffffffffa61fffff] [ 3.677566] debug: unmapping init [mem 0xffffffffa6914000-0xffffffffa69fffff] [ 3.713445] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.723706] systemd[1]: Detected virtualization kvm. [ 3.725978] systemd[1]: Detected architecture x86-64. [ 3.728720] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.756254] systemd[1]: No hostname configured. [ 3.758345] systemd[1]: Set hostname to . [ 3.760726] random: systemd: uninitialized urandom read (16 bytes read) [ 3.763393] systemd[1]: Initializing machine ID from random generator. [ 3.825315] random: ln: uninitialized urandom read (6 bytes read) [ 3.913778] random: systemd: uninitialized urandom read (16 bytes read) [ 3.916761] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.921792] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.933793] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.603265] device-mapper: uevent: version 1.0.3 [ 4.605251] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.290822] random: fast init done [ 5.319986] virtio_net virtio0 ens2: renamed from eth0 [ 5.339953] scsi host0: ata_piix [ 5.354592] scsi host1: ata_piix [ 5.356431] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.359460] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.008794] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.157960] random: crng init done [ 10.159667] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.572858] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.628491] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.874363] SELinux: Disabled at runtime. [ 11.938957] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.949380] systemd[1]: Detected virtualization kvm. [ 11.951649] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.434471] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.438481] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.443531] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.451234] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.454054] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.462228] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.467465] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 12.633826] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.928547] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.178490] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.198255] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.342273] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.364075] EDAC sbridge: Ver: 1.1.2 [ 15.076461] Key type dns_resolver registered [ 15.372189] NFS: Registering the id_resolver key type [ 15.374420] Key type id_resolver registered [ 15.376049] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg305-server login: [ 73.097010] hrtimer: interrupt took 3557161 ns [ 77.956351] libcfs: loading out-of-tree module taints kernel. [ 78.008425] Key type ._llcrypt registered [ 78.015673] Key type .llcrypt registered [ 78.184725] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 97.860418] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 99.216267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 99.232682] alg: No test for adler32 (adler32-zlib) [ 100.720134] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 101.441045] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 103.199238] Key type lgssc registered [ 104.967193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 123.982032] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 165.600900] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 180.252783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 180.315259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.761217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 181.804625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 181.948685] Lustre: lustre-MDT0000: new disk, initializing [ 182.071651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.094600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 186.918356] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.290770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.460193] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 200.538476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 200.546611] Lustre: Skipped 1 previous similar message [ 200.718260] Lustre: lustre-MDT0001: new disk, initializing [ 200.813956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.862210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 200.874856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 205.715884] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.795457] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 221.505338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.958541] Lustre: lustre-OST0000: new disk, initializing [ 221.964345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 221.980453] Lustre: 8448:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.106852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.255776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.274295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.338985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 229.911889] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.583374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.754350] Lustre: lustre-OST0001: new disk, initializing [ 247.757282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 247.761844] Lustre: 9522:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 247.858712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 254.126190] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.103405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 256.122367] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 256.167276] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 267.122120] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 272.842043] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 279.755392] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.070391] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing yml_node [ 290.601275] Lustre: DEBUG MARKER: Client: 2.17.58.41 [ 293.220954] Lustre: DEBUG MARKER: MDS: 2.17.58.41 [ 296.380399] Lustre: DEBUG MARKER: OSS: 2.17.58.41 [ 298.446145] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 14 19:30:19 EDT 2026 [ 316.727660] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 318.220446] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.403828] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.404940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.413433] Lustre: Skipped 3 previous similar messages [ 338.424774] Lustre: Skipped 3 previous similar messages [ 340.254566] Lustre: server umount lustre-MDT0000 complete [ 343.333990] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789428666 with bad export cookie 5797660026185034764 [ 343.339239] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.357476] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 343.520196] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.523636] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.542722] Lustre: Skipped 1 previous similar message [ 343.555823] Lustre: Skipped 1 previous similar message [ 348.640280] LustreError: 8455:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.640656] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.655646] LustreError: 8455:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 348.676143] Lustre: Skipped 1 previous similar message [ 349.704964] Lustre: server umount lustre-MDT0001 complete [ 353.422188] Lustre: server umount lustre-OST0000 complete [ 356.446820] Lustre: server umount lustre-OST0001 complete [ 361.774222] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 370.670791] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 407.152846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.397735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 407.428322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 407.535592] Lustre: lustre-MDT0000: new disk, initializing [ 407.628618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.642851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 411.506255] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 421.587466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.724025] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 421.791751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 421.804569] Lustre: Skipped 1 previous similar message [ 421.936281] Lustre: lustre-MDT0001: new disk, initializing [ 422.048596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 422.095205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 422.120731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.760733] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 434.169414] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 439.803276] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 446.298624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 446.767467] Lustre: lustre-OST0000: new disk, initializing [ 446.771451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 446.781820] Lustre: 17043:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 446.890460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.941842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 447.957946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 448.091466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 452.564456] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 462.115812] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 466.979553] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.238443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 471.105121] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 471.289648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 473.574160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.580695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.592772] LustreError: Skipped 1 previous similar message [ 473.594207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.621839] Lustre: Skipped 1 previous similar message [ 478.695682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 478.708569] Lustre: Skipped 2 previous similar messages [ 486.879391] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 487.138122] Lustre: server umount lustre-OST0000 complete [ 494.049898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.050875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 494.051597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 494.051602] Lustre: Skipped 2 previous similar messages [ 494.061386] Lustre: Skipped 1 previous similar message [ 496.637660] Lustre: server umount lustre-MDT0000 complete [ 499.169710] LustreError: 15086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.181510] LustreError: 15086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 499.791570] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789428822 with bad export cookie 5797660026185036297 [ 499.792516] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.811871] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 500.152776] Lustre: server umount lustre-MDT0001 complete [ 502.687390] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 19:33:44 (1789428824) [ 509.077844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.540622] LustreError: 19307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.604633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.693508] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 515.047906] LustreError: 19308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.163271] LustreError: 19307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.901963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.417460] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 532.263929] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 536.844155] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 542.202945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 547.913128] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 554.167516] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 566.247545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 566.251806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.257618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 566.257625] Lustre: Skipped 1 previous similar message [ 566.288074] Lustre: Skipped 1 previous similar message [ 571.286286] Lustre: server umount lustre-OST0000 complete [ 576.480871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.488431] Lustre: Skipped 1 previous similar message [ 580.749822] Lustre: server umount lustre-MDT0000 complete [ 581.605459] LustreError: 19308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.628388] LustreError: 19308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 584.511301] LustreError: 19289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789428907 with bad export cookie 5797660026185037179 [ 584.519168] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 584.533026] LustreError: 19289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 584.835783] Lustre: server umount lustre-MDT0001 complete [ 594.796824] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 597.069398] Key type lgssc unregistered [ 597.310728] LNet: 22836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.321491] LNetError: 22836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.335445] LNet: Removed LNI 192.168.203.105@tcp [ 598.308397] Key type .llcrypt unregistered [ 598.309902] Key type ._llcrypt unregistered [ 619.118386] Key type ._llcrypt registered [ 619.121618] Key type .llcrypt registered [ 619.229403] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 619.944983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 620.026865] alg: No test for adler32 (adler32-zlib) [ 621.042611] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 621.260598] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 622.960199] Key type lgssc registered [ 624.069897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 632.143903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 632.183554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.531871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.588204] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 644.074740] LustreError: 24237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.124764] LustreError: 24237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 648.092452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.820056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 653.397712] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 661.541760] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 667.443904] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 674.460107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 674.915830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 679.026206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 681.214808] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 688.311793] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 703.968762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 703.971488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.987838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 704.005575] Lustre: Skipped 1 previous similar message [ 704.013972] Lustre: Skipped 1 previous similar message [ 708.791759] Lustre: server umount lustre-OST0000 complete [ 714.209065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.222614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 714.230131] Lustre: Skipped 1 previous similar message [ 714.242761] Lustre: Skipped 1 previous similar message [ 718.499213] Lustre: server umount lustre-MDT0000 complete [ 719.338364] LustreError: 24237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 719.349880] LustreError: 24237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 722.034571] LustreError: 24218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789429044 with bad export cookie 7753760473110610842 [ 722.038330] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.051160] LustreError: 24218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 722.544861] Lustre: server umount lustre-MDT0001 complete [ 733.275360] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 736.112813] Key type lgssc unregistered [ 736.487741] LNet: 28214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.497365] LNetError: 28214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.518127] LNet: Removed LNI 192.168.203.105@tcp [ 737.386292] Key type .llcrypt unregistered [ 737.388957] Key type ._llcrypt unregistered [ 760.505567] Key type ._llcrypt registered [ 760.510935] Key type .llcrypt registered [ 760.696945] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 761.622540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 761.648987] alg: No test for adler32 (adler32-zlib) [ 762.694978] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 762.977732] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 764.681120] Key type lgssc registered [ 766.231138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 773.756435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 773.782303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.188613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.631482] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 784.354070] LustreError: 29627:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 784.379116] LustreError: 29627:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 787.909565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.344226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 792.957246] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 800.776764] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 806.468855] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 820.338362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 820.679759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 826.934539] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 833.351419] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 844.273762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 849.378380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.383211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.397641] Lustre: Skipped 1 previous similar message [ 851.395311] Lustre: server umount lustre-OST0000 complete [ 855.012133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 855.022900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.048573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 855.058463] Lustre: Skipped 1 previous similar message [ 859.616521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 859.617454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.739341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 864.752514] Lustre: Skipped 2 previous similar messages [ 869.343134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 869.512713] Lustre: server umount lustre-MDT0000 complete [ 869.860992] LustreError: 29622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.906376] LustreError: 29622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 872.948637] LustreError: 29608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789429195 with bad export cookie 1428571907421214327 [ 872.955047] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 872.964295] LustreError: 29608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 873.433715] Lustre: server umount lustre-MDT0001 complete [ 883.513911] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 886.370427] Key type lgssc unregistered [ 886.644184] LNet: 33526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.655717] LNetError: 33526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.675064] LNet: Removed LNI 192.168.203.105@tcp [ 887.539000] Key type .llcrypt unregistered [ 887.541425] Key type ._llcrypt unregistered [ 909.511626] Key type ._llcrypt registered [ 909.513424] Key type .llcrypt registered [ 909.595537] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 910.681242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 910.699461] alg: No test for adler32 (adler32-zlib) [ 911.883598] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 912.160708] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 913.873395] Key type lgssc registered [ 915.265776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 924.057698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 924.102248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.817666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.333351] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 935.396386] LustreError: 34937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.416473] LustreError: 34937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 938.116318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.456619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 942.500246] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 949.174275] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 954.094534] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 960.677706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 961.101684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.408581] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 970.353118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 974.377840] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 984.035301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 984.037721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.049973] LustreError: Skipped 1 previous similar message [ 984.053469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.067253] Lustre: Skipped 1 previous similar message [ 984.083481] Lustre: Skipped 1 previous similar message [ 986.087194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.094982] Lustre: Skipped 1 previous similar message [ 991.205891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.215610] Lustre: Skipped 1 previous similar message [ 996.319384] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 996.323495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.333360] Lustre: Skipped 1 previous similar message [ 996.446613] Lustre: server umount lustre-OST0000 complete [ 999.904615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 999.913560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.444286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1001.448364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.448744] Lustre: Skipped 1 previous similar message [ 1011.680532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1011.686396] Lustre: Skipped 4 previous similar messages [ 1014.241054] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1014.592428] Lustre: server umount lustre-MDT0000 complete [ 1016.807387] LustreError: 37428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1016.825439] LustreError: 37428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1018.333057] LustreError: 34918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789429341 with bad export cookie 17291584208773565077 [ 1018.333768] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1018.341025] LustreError: 34918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1018.659154] Lustre: server umount lustre-MDT0001 complete [ 1028.216238] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1030.840325] Key type lgssc unregistered [ 1031.107721] LNet: 38472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.115677] LNetError: 38472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.127179] LNet: Removed LNI 192.168.203.105@tcp [ 1031.834758] Key type .llcrypt unregistered [ 1031.837062] Key type ._llcrypt unregistered [ 1052.514328] Key type ._llcrypt registered [ 1052.517480] Key type .llcrypt registered [ 1052.605652] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1053.678356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1053.701359] alg: No test for adler32 (adler32-zlib) [ 1054.726668] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 1054.939592] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1056.592096] Key type lgssc registered [ 1057.396553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1063.793242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1063.811662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.272830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.558923] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1075.685585] LustreError: 39873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.708958] LustreError: 39873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1077.221786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.598162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1082.052777] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1089.706834] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1094.954298] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1102.129116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1102.373418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.533915] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1110.515066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1116.242025] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1122.786848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1122.806865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.827831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.857742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.871545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1125.879780] Lustre: Skipped 1 previous similar message [ 1125.904603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.175377] LustreError: 41695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.332914] Lustre: server umount lustre-OST0000 complete [ 1149.420608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1149.435906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.477706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1149.482578] Lustre: Skipped 1 previous similar message [ 1155.196070] Lustre: server umount lustre-MDT0000 complete [ 1156.576745] LustreError: 39870:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.621104] LustreError: 39870:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1159.988893] LustreError: 39855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789429482 with bad export cookie 7048819353812950233 [ 1159.991566] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.011203] LustreError: 39855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1160.046036] LustreError: 39184:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 1166.337669] Lustre: server umount lustre-MDT0001 complete [ 1175.621768] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1178.045846] Key type lgssc unregistered [ 1178.360150] LNet: 43581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.366792] LNetError: 43581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.388627] LNet: Removed LNI 192.168.203.105@tcp [ 1179.247291] Key type .llcrypt unregistered [ 1179.249702] Key type ._llcrypt unregistered [ 1202.033730] Key type ._llcrypt registered [ 1202.037669] Key type .llcrypt registered [ 1202.152225] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1203.117382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1203.191474] alg: No test for adler32 (adler32-zlib) [ 1204.276876] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 1204.471715] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1206.127368] Key type lgssc registered [ 1207.440660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1214.473542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1214.497781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.962537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1220.353509] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1225.191544] LustreError: 44990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1225.217626] LustreError: 44990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1228.529760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.931788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1233.417461] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1240.792612] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1246.287887] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1252.720663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1253.122110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1259.597081] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1260.278730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1267.496833] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1274.636701] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.5@tcp (stopping) [ 1274.848819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1274.857476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.873563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1275.878331] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.883985] Lustre: Skipped 2 previous similar messages [ 1275.928930] Lustre: Skipped 1 previous similar message [ 1279.734632] LustreError: 47164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1279.902452] Lustre: server umount lustre-MDT0000 complete [ 1280.998914] LustreError: 44991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.043297] LustreError: 44991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1284.859000] LustreError: 47568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789429607 with bad export cookie 2861410337406877081 [ 1284.866837] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1284.873822] LustreError: 47568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1284.885153] LustreError: 44985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.961278] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.5@tcp (stopping) [ 1284.971691] Lustre: Skipped 1 previous similar message [ 1286.128459] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.125507] LustreError: 44986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.127300] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1289.142449] LustreError: 44986:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1289.171043] Lustre: Skipped 1 previous similar message [ 1291.389932] Lustre: server umount lustre-MDT0001 complete [ 1312.817734] Lustre: server umount lustre-OST0000 complete [ 1327.678449] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1330.256648] Key type lgssc unregistered [ 1330.520923] LNet: 48850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.530988] LNetError: 48850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.558645] LNet: Removed LNI 192.168.203.105@tcp [ 1331.387289] Key type .llcrypt unregistered [ 1331.390766] Key type ._llcrypt unregistered [ 1354.714190] Key type ._llcrypt registered [ 1354.718980] Key type .llcrypt registered [ 1354.825803] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1355.866520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1355.920878] alg: No test for adler32 (adler32-zlib) [ 1357.198690] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 1357.465744] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1359.127154] Key type lgssc registered [ 1360.448788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1367.901320] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1367.936750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1383.584241] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1388.703927] LustreError: 50236:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 1415.391676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.480836] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1428.550831] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1593.941490] Lustre: server umount lustre-OST0000 complete [ 1608.502955] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1610.897621] Key type lgssc unregistered [ 1611.126353] LNet: 51907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.136763] LNetError: 51907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.169316] LNet: Removed LNI 192.168.203.105@tcp [ 1611.892563] Key type .llcrypt unregistered [ 1611.897730] Key type ._llcrypt unregistered [ 1631.560351] Key type ._llcrypt registered [ 1631.562324] Key type .llcrypt registered [ 1631.658781] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1632.459946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1632.602080] alg: No test for adler32 (adler32-zlib) [ 1633.611289] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 1633.843175] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1635.495240] Key type lgssc registered [ 1636.421793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1643.586603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1643.606556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.143991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.868272] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1654.243894] LustreError: 53298:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.267235] LustreError: 53298:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1657.846221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.227762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1663.162462] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1671.426590] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1676.553610] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1684.289505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1684.731130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.854343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1690.413560] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1697.279920] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1700.324067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1700.333042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1700.344550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1702.456636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1707.493569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1707.500665] Lustre: Skipped 1 previous similar message [ 1712.617032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1712.619733] Lustre: Skipped 1 previous similar message [ 1714.143169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1714.399803] Lustre: server umount lustre-OST0000 complete [ 1719.776424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1719.794919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1719.824058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1719.829344] Lustre: Skipped 1 previous similar message [ 1722.856304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.989931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1728.008985] Lustre: Skipped 3 previous similar messages [ 1733.087442] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1733.343000] Lustre: server umount lustre-MDT0000 complete [ 1737.842507] LustreError: 53280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789430060 with bad export cookie 15675235574807150886 [ 1737.844698] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1737.858314] LustreError: 53280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1738.567461] Lustre: server umount lustre-MDT0001 complete [ 1752.971732] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1756.055253] Key type lgssc unregistered [ 1756.508871] LNet: 56833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1756.526166] LNetError: 56833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1756.566040] LNet: Removed LNI 192.168.203.105@tcp [ 1757.459178] Key type .llcrypt unregistered [ 1757.464339] Key type ._llcrypt unregistered [ 1780.710869] Key type ._llcrypt registered [ 1780.713649] Key type .llcrypt registered [ 1780.804863] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1781.703359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1781.821433] alg: No test for adler32 (adler32-zlib) [ 1782.835273] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 1783.132816] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1784.815199] Key type lgssc registered [ 1785.762691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1793.276618] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1793.311926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1808.927391] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1814.047584] LustreError: 58214:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 1840.735285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.616952] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1854.611589] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1862.041734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.250358] LustreError: 57549:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005067407f x1876353461782784/t0(0) o250->MGC192.168.203.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1862.756726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.873720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1867.416921] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1867.748931] LustreError: 58239:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1867.783407] LustreError: 58239:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1872.872741] LustreError: 58240:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1875.943451] LustreError: 59252:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1875.967161] LustreError: 59252:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1876.703975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.120461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1881.106991] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1887.496833] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1892.127892] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1897.440679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1897.450173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.481712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1897.960511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.249740] Lustre: server umount lustre-OST0000 complete [ 1922.528182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.530558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1922.550884] Lustre: Skipped 1 previous similar message [ 1922.558531] Lustre: Skipped 1 previous similar message [ 1927.588269] Lustre: server umount lustre-MDT0000 complete [ 1927.650800] LustreError: 59942:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1927.666766] LustreError: 59942:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1930.936718] LustreError: 58220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789430253 with bad export cookie 15888336715497963450 [ 1930.948676] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1930.951239] LustreError: 58220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1937.406586] Lustre: server umount lustre-MDT0001 complete [ 1948.609816] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 1951.572508] Key type lgssc unregistered [ 1951.867366] LNet: 61955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.882940] LNetError: 61955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.934464] LNet: Removed LNI 192.168.203.105@tcp [ 1953.769876] Key type .llcrypt unregistered [ 1953.772326] Key type ._llcrypt unregistered [ 1976.539548] Key type ._llcrypt registered [ 1976.546117] Key type .llcrypt registered [ 1976.640806] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 1977.616173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.628108] alg: No test for adler32 (adler32-zlib) [ 1978.652959] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 1978.903160] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 1980.591220] Key type lgssc registered [ 1981.579993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1987.860678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1987.900378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.301130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.813904] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 1999.846740] LustreError: 63372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.874063] LustreError: 63372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2000.559694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.933159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2004.842892] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2011.139646] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2016.360789] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2022.331408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2022.706375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.818326] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2035.003547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2036.136239] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2055.652887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.662526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2055.674571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2055.898039] Lustre: server umount lustre-OST0000 complete [ 2060.770972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.786830] Lustre: Skipped 1 previous similar message [ 2060.794986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.807619] Lustre: Skipped 2 previous similar messages [ 2065.145984] Lustre: server umount lustre-MDT0000 complete [ 2065.890319] LustreError: 63367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.909906] LustreError: 63367:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2068.221922] LustreError: 63353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789430391 with bad export cookie 9359087117771910449 [ 2068.225945] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2068.238421] LustreError: 63353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2068.675984] Lustre: server umount lustre-MDT0001 complete [ 2078.037960] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2080.437852] Key type lgssc unregistered [ 2080.711164] LNet: 66925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.718287] LNetError: 66925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.737241] LNet: Removed LNI 192.168.203.105@tcp [ 2081.455442] Key type .llcrypt unregistered [ 2081.458292] Key type ._llcrypt unregistered [ 2103.987872] Key type ._llcrypt registered [ 2103.992376] Key type .llcrypt registered [ 2104.108678] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2105.131312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2105.376293] alg: No test for adler32 (adler32-zlib) [ 2106.434531] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 2106.687405] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2108.415229] Key type lgssc registered [ 2109.560622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2118.086447] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 20:00:39 (1789430439) [ 2124.383838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2124.420821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.898501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.458134] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2136.037987] LustreError: 68618:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.062965] LustreError: 68618:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2138.595846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.930601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2143.510550] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2150.470427] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2155.885449] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2162.690278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.116162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.804980] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2170.284616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2177.242210] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2183.577516] Lustre: Failing over lustre-MDT0000 [ 2183.928606] Lustre: server umount lustre-MDT0000 complete [ 2185.009206] LustreError: 68615:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.190037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2185.200888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.067095] LustreError: 68613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2190.082128] LustreError: 68613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2193.255786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.493217] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2193.635369] Lustre: *** cfs_fail_loc=135, val=0*** [ 2193.738309] LustreError: 71719:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2193.750168] LustreError: 71719:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.105@tcp: cfg command failed: rc = -2 [ 2193.761136] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2193.761136] [ 2193.773036] LustreError: MGC192.168.203.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2193.794862] LustreError: 71707:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2193.808947] LustreError: 71707:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2193.822136] LustreError: 71707:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2193.839409] LustreError: 71707:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2193.872559] Lustre: server umount lustre-MDT0000 complete [ 2193.874513] LustreError: 71707:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2195.193898] LustreError: 71122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.214785] LustreError: 71122:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2199.695370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.831565] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2200.069979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.139596] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2200.309302] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2204.421361] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2205.159597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2205.189859] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2205.246307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2215.398249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2215.401413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.405639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2215.413489] Lustre: Skipped 3 previous similar messages [ 2215.429558] Lustre: Skipped 1 previous similar message [ 2217.355311] Lustre: server umount lustre-OST0000 complete [ 2225.634883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.638855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2225.643148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2225.658750] Lustre: Skipped 1 previous similar message [ 2230.753477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2230.756751] Lustre: Skipped 2 previous similar messages [ 2235.359351] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2235.514789] Lustre: server umount lustre-MDT0000 complete [ 2235.875284] LustreError: 68619:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.898805] LustreError: 68619:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2238.601619] LustreError: 68600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789430561 with bad export cookie 6423896618507817846 [ 2238.605760] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2238.613547] LustreError: 68600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2238.886319] Lustre: server umount lustre-MDT0001 complete [ 2248.013243] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2250.196577] Key type lgssc unregistered [ 2250.476120] LNet: 73498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.484642] LNetError: 73498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2250.500726] LNet: Removed LNI 192.168.203.105@tcp [ 2251.335579] Key type .llcrypt unregistered [ 2251.338247] Key type ._llcrypt unregistered [ 2271.064647] Key type ._llcrypt registered [ 2271.066943] Key type .llcrypt registered [ 2271.193351] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2272.306778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2272.351444] alg: No test for adler32 (adler32-zlib) [ 2273.447499] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 2273.689939] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2275.351287] Key type lgssc registered [ 2276.596513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2283.821748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2283.837597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2285.273979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2290.322737] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2295.783972] LustreError: 74886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.825569] LustreError: 74886:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2299.586407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2299.977339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2304.520261] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2311.448744] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2316.683654] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2322.264252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2322.531746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.949132] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2330.734794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2334.416540] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2340.317104] Lustre: Failing over lustre-MDT0000 [ 2340.471471] Lustre: server umount lustre-MDT0000 complete [ 2340.833068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2340.854734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.872802] LustreError: 74886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.360431] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.375017] Lustre: Skipped 1 previous similar message [ 2342.166312] LustreError: 74880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.200201] LustreError: 74880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2346.481498] LustreError: 74886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.513484] LustreError: 74886:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2350.416683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.539664] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2350.622194] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2350.743582] LustreError: 77974:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2350.763719] LustreError: 77974:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.105@tcp: cfg command failed: rc = -9 [ 2350.780971] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2350.780971] [ 2350.797183] LustreError: MGC192.168.203.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2350.812630] LustreError: 77962:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2350.829484] LustreError: 77962:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2350.847489] LustreError: 77962:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2350.858202] LustreError: 77962:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2350.922861] Lustre: server umount lustre-MDT0000 complete [ 2350.930378] LustreError: 77962:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2351.584179] LustreError: 74887:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.606931] LustreError: 74887:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2357.466409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.570902] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2357.583602] LustreError: 74877:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2357.910376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2357.981698] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2358.445794] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2362.493790] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2363.372064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2363.405865] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2363.464431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2369.000676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2369.010056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2369.039819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2371.040827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2371.049813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.065634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2373.602232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2378.724103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2378.737530] Lustre: Skipped 1 previous similar message [ 2383.327130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2383.502290] Lustre: server umount lustre-OST0000 complete [ 2388.964685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.967302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2388.974215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2388.995468] Lustre: Skipped 2 previous similar messages [ 2393.474803] Lustre: server umount lustre-MDT0000 complete [ 2394.087446] LustreError: 74881:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.117601] LustreError: 74881:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2397.266679] LustreError: 74865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789430720 with bad export cookie 6824526864741153363 [ 2397.272779] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.284521] LustreError: 74865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2397.587720] Lustre: server umount lustre-MDT0001 complete [ 2408.705996] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2411.157818] Key type lgssc unregistered [ 2411.480875] LNet: 79753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2411.490817] LNetError: 79753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2411.506182] LNet: Removed LNI 192.168.203.105@tcp [ 2412.367771] Key type .llcrypt unregistered [ 2412.369671] Key type ._llcrypt unregistered [ 2424.368613] Key type ._llcrypt registered [ 2424.371265] Key type .llcrypt registered [ 2424.510595] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2424.797389] Key type .llcrypt unregistered [ 2424.799234] Key type ._llcrypt unregistered [ 2445.557238] Key type ._llcrypt registered [ 2445.560875] Key type .llcrypt registered [ 2445.662550] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2446.673987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2446.737803] alg: No test for adler32 (adler32-zlib) [ 2447.920024] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 2448.200836] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2449.912516] Key type lgssc registered [ 2451.036501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2459.981884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2460.009946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.596666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.386853] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2471.905598] LustreError: 81628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.942391] LustreError: 81628:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2474.631070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.008501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2478.267506] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2483.941896] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2488.199534] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2493.591235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2493.891740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.905473] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2506.512790] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2507.190208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2520.544384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2520.556502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2520.571255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.592689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2522.596545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2522.603695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.971863] Lustre: server umount lustre-OST0000 complete [ 2531.296213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2531.300058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.332925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2531.340535] Lustre: Skipped 1 previous similar message [ 2536.473583] Lustre: server umount lustre-MDT0000 complete [ 2537.956519] LustreError: 81625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.992759] LustreError: 81625:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2539.847178] LustreError: 81610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789430862 with bad export cookie 14392756825848243815 [ 2539.856608] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.867480] LustreError: 81610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2540.522985] Lustre: server umount lustre-MDT0001 complete [ 2551.171452] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2554.019453] Key type lgssc unregistered [ 2554.396246] LNet: 85178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2554.401609] LNetError: 85178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2554.431682] LNet: Removed LNI 192.168.203.105@tcp [ 2555.734097] Key type .llcrypt unregistered [ 2555.736997] Key type ._llcrypt unregistered [ 2576.992315] Key type ._llcrypt registered [ 2576.994488] Key type .llcrypt registered [ 2577.093269] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2577.963042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2578.080934] alg: No test for adler32 (adler32-zlib) [ 2579.091451] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 2579.275859] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2580.927199] Key type lgssc registered [ 2582.005417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2588.771288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2588.804510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.312398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2594.134959] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2600.435314] LustreError: 86565:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.458036] LustreError: 86565:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2603.029874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.435376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2608.716553] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2616.943917] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2623.496650] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2630.172554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2630.586658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.716160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2636.635536] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2643.377138] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2654.184499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.186821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2654.187358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2654.194456] Lustre: Skipped 1 previous similar message [ 2659.227971] Lustre: server umount lustre-OST0000 complete [ 2664.416903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.431411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2664.434332] Lustre: Skipped 1 previous similar message [ 2668.878256] Lustre: server umount lustre-MDT0000 complete [ 2669.541262] LustreError: 86566:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.583601] LustreError: 86566:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2672.238193] LustreError: 86547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789430995 with bad export cookie 8239687485129715506 [ 2672.248344] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2672.535328] Lustre: server umount lustre-MDT0001 complete [ 2682.263110] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2684.959650] Key type lgssc unregistered [ 2685.322801] LNet: 90097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2685.343594] LNetError: 90097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2685.370665] LNet: Removed LNI 192.168.203.105@tcp [ 2686.249353] Key type .llcrypt unregistered [ 2686.253422] Key type ._llcrypt unregistered [ 2706.679504] Key type ._llcrypt registered [ 2706.681797] Key type .llcrypt registered [ 2706.783799] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2707.684728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2707.714512] alg: No test for adler32 (adler32-zlib) [ 2708.774734] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 2708.973628] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2710.639321] Key type lgssc registered [ 2711.909299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2718.875789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2718.895641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.456136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.736940] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2730.977304] LustreError: 91486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.992550] LustreError: 91486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2732.150242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.518238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2736.699477] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2744.319988] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2749.680746] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2755.853719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2756.126172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.135477] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2765.423306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2768.899464] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2781.160692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2781.168626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.180968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2781.189767] Lustre: Skipped 1 previous similar message [ 2781.209464] Lustre: Skipped 1 previous similar message [ 2783.380649] Lustre: server umount lustre-OST0000 complete [ 2788.833158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2788.842090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.862729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2791.392745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2791.397659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2796.513376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2796.521070] Lustre: Skipped 2 previous similar messages [ 2801.119983] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2801.327626] Lustre: server umount lustre-MDT0000 complete [ 2801.634566] LustreError: 91486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.652634] LustreError: 91486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2805.090427] LustreError: 91467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789431127 with bad export cookie 14055018756902967607 [ 2805.091679] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2805.099020] LustreError: 91467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2805.488556] Lustre: server umount lustre-MDT0001 complete [ 2816.486203] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 2819.378254] Key type lgssc unregistered [ 2819.727131] LNet: 95035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2819.747262] LNetError: 95035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2819.778754] LNet: Removed LNI 192.168.203.105@tcp [ 2820.680968] Key type .llcrypt unregistered [ 2820.684253] Key type ._llcrypt unregistered [ 2843.318157] Key type ._llcrypt registered [ 2843.321899] Key type .llcrypt registered [ 2843.478597] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 2844.347646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2844.539067] alg: No test for adler32 (adler32-zlib) [ 2845.620684] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 2845.833686] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 2847.523341] Key type lgssc registered [ 2848.806150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2855.986149] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2856.012582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2871.584853] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2876.707148] LustreError: 96416:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 2903.392467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2909.461744] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2916.509970] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2922.515711] Lustre: server umount lustre-OST0000 complete [ 2930.897654] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 20:14:12 (1789431252) [ 2937.710262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.035734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.069934] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2943.466040] LustreError: 98053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2943.481694] LustreError: 98053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2947.558687] LustreError: 98054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.799289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.169492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2954.443195] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2961.580957] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2966.779461] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2972.949984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2973.307797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.825690] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 2982.585660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2987.425174] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3000.289302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3000.298866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.320354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3003.360523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3003.362788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3003.374289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.855829] Lustre: server umount lustre-OST0000 complete [ 3011.553062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3011.562870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.579323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.586908] Lustre: Skipped 1 previous similar message [ 3013.602647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.609119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3013.643662] Lustre: Skipped 1 previous similar message [ 3014.962801] Lustre: server umount lustre-MDT0000 complete [ 3018.547662] LustreError: 98035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789431341 with bad export cookie 485895127936717009 [ 3018.567667] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3018.573714] LustreError: 98035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3018.958593] Lustre: server umount lustre-MDT0001 complete [ 3029.868494] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3032.557758] Key type lgssc unregistered [ 3032.847510] LNet: 101746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3032.855437] LNetError: 101746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3032.880621] LNet: Removed LNI 192.168.203.105@tcp [ 3033.732380] Key type .llcrypt unregistered [ 3033.734357] Key type ._llcrypt unregistered [ 3075.457406] Key type ._llcrypt registered [ 3075.461582] Key type .llcrypt registered [ 3075.739817] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3076.948973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3077.011781] alg: No test for adler32 (adler32-zlib) [ 3078.362160] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 3078.786257] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3080.528181] Key type lgssc registered [ 3082.003170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3090.481673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3090.508168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.057207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3096.538465] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3101.167159] LustreError: 103927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3101.202038] LustreError: 103927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3105.735505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.230985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3111.745448] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3119.585453] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3125.771437] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3133.806640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3134.377278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3136.448587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3141.446822] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3149.745376] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3161.057509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3161.069968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.073244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3161.085719] Lustre: Skipped 1 previous similar message [ 3161.103961] Lustre: Skipped 1 previous similar message [ 3165.595563] Lustre: server umount lustre-OST0000 complete [ 3171.296336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.299223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3171.312230] Lustre: Skipped 1 previous similar message [ 3175.321659] Lustre: server umount lustre-MDT0000 complete [ 3176.416640] LustreError: 103921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.446698] LustreError: 103921:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3179.357059] LustreError: 103908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789431502 with bad export cookie 6655993226168111383 [ 3179.373092] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3179.861286] Lustre: server umount lustre-MDT0001 complete [ 3190.457393] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3193.165761] Key type lgssc unregistered [ 3193.518492] LNet: 107458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3193.526209] LNetError: 107458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3193.541939] LNet: Removed LNI 192.168.203.105@tcp [ 3194.389219] Key type .llcrypt unregistered [ 3194.392380] Key type ._llcrypt unregistered [ 3213.268608] Key type ._llcrypt registered [ 3213.271308] Key type .llcrypt registered [ 3213.398645] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3214.187863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3214.228386] alg: No test for adler32 (adler32-zlib) [ 3215.427405] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 3215.746301] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3217.399616] Key type lgssc registered [ 3218.693343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3226.361761] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3226.383344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3241.954115] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3247.071343] LustreError: 108697:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 3273.695161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3275.755642] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3278.195266] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3280.448074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3280.544462] LustreError: MGC192.168.203.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3280.548958] LustreError: 109718:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3280.551608] LustreError: 109718:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3280.554962] LustreError: 109718:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3280.586653] Lustre: server umount lustre-MDT0000 complete [ 3280.588667] LustreError: 109718:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3292.682556] Lustre: server umount lustre-OST0000 complete [ 3295.855369] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 3299.366134] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3312.084664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.176205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3312.188254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3312.231548] Lustre: lustre-MDT0000: new disk, initializing [ 3312.256829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3312.263721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3313.765843] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3318.009261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.051084] Lustre: 112505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3318.077786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3318.080367] Lustre: Skipped 1 previous similar message [ 3318.117069] Lustre: lustre-MDT0001: new disk, initializing [ 3318.151939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3318.164487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3318.169183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3319.574699] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3323.324054] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3325.127126] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3327.333743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.441104] Lustre: lustre-OST0000: new disk, initializing [ 3327.442735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3327.445123] Lustre: 114473:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3327.476503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3329.392842] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3333.039903] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3334.476696] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3337.215545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3337.219498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3337.255764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3337.623086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3339.161651] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3339.254419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3342.305988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3342.307970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.309830] LustreError: Skipped 1 previous similar message [ 3342.310453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3342.318540] Lustre: Skipped 1 previous similar message [ 3346.071386] Lustre: server umount lustre-OST0000 complete [ 3348.960552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3348.964566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.971045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3348.973795] Lustre: Skipped 1 previous similar message [ 3352.544714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3352.545417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3353.625465] Lustre: server umount lustre-MDT0000 complete [ 3354.814223] LustreError: 112497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789431677 with bad export cookie 11281562207423809985 [ 3354.818361] LustreError: 112497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3354.818718] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3354.943473] Lustre: server umount lustre-MDT0001 complete [ 3357.642650] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 20:21:20 (1789431680) [ 3358.716766] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3363.827428] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 3366.915580] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3378.133895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.216172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3378.226374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3378.265120] Lustre: lustre-MDT0000: new disk, initializing [ 3378.285540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.291256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3379.664713] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3384.145980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.193108] Lustre: 119383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3384.216561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3384.218916] Lustre: Skipped 1 previous similar message [ 3384.254198] Lustre: lustre-MDT0001: new disk, initializing [ 3384.312701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3384.316727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3385.796504] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3389.664428] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3391.596144] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3393.863176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3393.970804] Lustre: lustre-OST0000: new disk, initializing [ 3393.972596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3393.975391] Lustre: 121351:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3394.012166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3394.014848] Lustre: Skipped 1 previous similar message [ 3395.877770] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3399.488084] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3400.697356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3400.700904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3400.740195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3400.825785] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3400.892270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3402.152815] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3402.215728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3405.793056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3405.794098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.795875] LustreError: Skipped 1 previous similar message [ 3405.800065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3405.803904] Lustre: Skipped 2 previous similar messages [ 3408.916928] Lustre: server umount lustre-OST0000 complete [ 3410.913195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.913669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3410.916642] Lustre: Skipped 2 previous similar messages [ 3416.215630] Lustre: server umount lustre-MDT0000 complete [ 3417.418833] LustreError: 119377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789431740 with bad export cookie 11281562207423810853 [ 3417.420826] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3417.422976] LustreError: 119377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3417.536376] Lustre: server umount lustre-MDT0001 complete [ 3420.085259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.232948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.419345] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3424.085733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3425.414183] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3427.616963] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3429.220557] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3431.232808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3433.104400] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3435.428915] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3448.478218] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3449.824367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3449.827632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.832584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3449.835134] Lustre: Skipped 3 previous similar messages [ 3451.534770] Lustre: server umount lustre-OST0000 complete [ 3454.944638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3454.946916] LustreError: Skipped 1 previous similar message [ 3458.835421] Lustre: server umount lustre-MDT0000 complete [ 3459.963054] LustreError: 125405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789431782 with bad export cookie 11281562207423811721 [ 3459.965190] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3459.966623] LustreError: 125405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3460.078414] Lustre: server umount lustre-MDT0001 complete [ 3463.301679] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 3464.215782] Key type lgssc unregistered [ 3464.338320] LNet: 127210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3464.341790] LNetError: 127210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3464.348293] LNet: Removed LNI 192.168.203.105@tcp [ 3464.653832] Key type .llcrypt unregistered [ 3464.654939] Key type ._llcrypt unregistered [ 3469.985446] Key type ._llcrypt registered [ 3469.986463] Key type .llcrypt registered [ 3470.020127] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 3475.740369] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 3476.139310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3476.149186] alg: No test for adler32 (adler32-zlib) [ 3476.993455] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 3477.082745] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 3478.663212] Key type lgssc registered [ 3479.008792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3489.157444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3489.162222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.238289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3490.247240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3490.279164] Lustre: lustre-MDT0000: new disk, initializing [ 3490.299537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3490.306842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3491.537403] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3495.322673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3495.356815] Lustre: 130332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3495.382261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3495.384484] Lustre: Skipped 1 previous similar message [ 3495.423797] Lustre: lustre-MDT0001: new disk, initializing [ 3495.455110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3495.467105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3495.471242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3496.838301] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3500.447973] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3502.223520] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3504.320557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3504.418309] Lustre: lustre-OST0000: new disk, initializing [ 3504.420623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3504.424562] Lustre: 132299:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3504.455618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3506.265721] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3508.276060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3508.279545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3508.308831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3509.822709] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3511.186612] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3511.256425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3512.517592] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3512.580946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3513.313728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3513.314385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.315279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3513.317815] LustreError: Skipped 1 previous similar message [ 3513.322550] Lustre: Skipped 1 previous similar message [ 3515.215710] Lustre: server umount lustre-OST0000 complete [ 3518.432584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.433231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3518.436258] Lustre: Skipped 1 previous similar message [ 3518.437965] Lustre: Skipped 1 previous similar message [ 3522.452432] Lustre: server umount lustre-MDT0000 complete [ 3523.640040] LustreError: 130323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789431846 with bad export cookie 1916095918648959002 [ 3523.642547] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3523.643260] LustreError: 130323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3523.755567] Lustre: server umount lustre-MDT0001 complete [ 3526.196622] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 20:24:08 (1789431848) [ 3528.174985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.330784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.469528] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3532.042070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.336717] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3535.575996] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3537.248346] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3541.984509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.989770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3541.992522] Lustre: Skipped 2 previous similar messages [ 3547.104488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3547.106913] Lustre: Skipped 1 previous similar message [ 3552.223105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3552.224560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3552.228327] Lustre: Skipped 1 previous similar message [ 3552.273438] Lustre: server umount lustre-MDT0000 complete [ 3553.432061] LustreError: 134733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789431876 with bad export cookie 1916095918648959870 [ 3553.435651] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3553.435948] LustreError: 134733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3553.538190] Lustre: server umount lustre-MDT0001 complete [ 3556.195923] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 20:24:38 (1789431878) [ 3558.274358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3573.727461] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3578.847335] LustreError: 137119:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 3604.447287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3604.449307] Lustre: Skipped 1 previous similar message [ 3606.245622] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3608.511349] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3609.183570] Lustre: server umount lustre-OST0000 complete [ 3611.618101] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 20:25:34 (1789431934) [ 3613.582518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3613.728487] LustreError: 138562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3613.735718] LustreError: 138562:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3613.752448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.067443] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3617.923142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.527544] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3621.846650] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3623.507526] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3625.542152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.404373] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3629.547627] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3643.872122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3643.874930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.879046] Lustre: Skipped 1 previous similar message [ 3643.881492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3647.711807] Lustre: server umount lustre-MDT0000 complete [ 3648.919257] LustreError: 138545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789431971 with bad export cookie 1916095918648960507 [ 3648.922114] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3648.923203] LustreError: 138545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3649.027765] Lustre: server umount lustre-MDT0001 complete [ 3660.438198] Lustre: server umount lustre-OST0000 complete [ 3663.270903] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 20:26:25 (1789431985) [ 3665.455190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3665.599822] LustreError: 142230:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.626097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3665.628594] Lustre: Skipped 2 previous similar messages [ 3667.035530] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3669.901938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3671.400774] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3673.761725] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3675.523817] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3677.599651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3679.472054] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3681.683367] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3682.937052] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3691.046942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3691.143433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3692.525642] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3692.597105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3695.071816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3695.075108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3695.079795] Lustre: Skipped 2 previous similar messages [ 3695.081466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3695.083473] Lustre: Skipped 3 previous similar messages [ 3699.342427] Lustre: server umount lustre-OST0000 complete [ 3700.703738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3700.705756] LustreError: Skipped 1 previous similar message [ 3706.644435] Lustre: server umount lustre-MDT0000 complete [ 3707.810125] LustreError: 144073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432030 with bad export cookie 1916095918648961564 [ 3707.812986] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3707.814873] LustreError: 144073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3707.920531] Lustre: server umount lustre-MDT0001 complete [ 3710.431856] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 20:27:13 (1789432033) [ 3712.415731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3727.839501] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3732.959493] LustreError: 146453:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 3758.559317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.561662] Lustre: Skipped 2 previous similar messages [ 3760.265860] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3762.398103] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3764.484526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3764.615560] LustreError: 146478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3764.659166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3765.833335] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3768.492781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.887722] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3772.135101] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3773.886180] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3775.185820] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3775.249736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3776.462878] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3776.528513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3779.041260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3779.041587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3779.042710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3779.042717] Lustre: Skipped 7 previous similar messages [ 3779.046051] Lustre: Skipped 5 previous similar messages [ 3783.258056] Lustre: server umount lustre-MDT0000 complete [ 3784.161522] LustreError: 147486:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3784.165650] LustreError: 147486:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3784.433043] LustreError: 146460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432107 with bad export cookie 1916095918648962572 [ 3784.435179] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3784.437982] LustreError: 146460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3784.540392] Lustre: server umount lustre-MDT0001 complete [ 3795.860661] Lustre: server umount lustre-OST0000 complete [ 3798.263978] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 20:28:40 (1789432120) [ 3800.324993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3815.775594] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3820.895432] LustreError: 150640:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 3846.495401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3846.498684] Lustre: Skipped 2 previous similar messages [ 3848.250251] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3850.431668] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3852.444813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.570728] LustreError: 150665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3852.616495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3853.823157] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3856.399159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3857.781962] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3860.029815] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3861.799963] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3863.936053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3864.000853] Lustre: lustre-OST0001: new disk, initializing [ 3864.002558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3864.005052] Lustre: 153513:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3865.725306] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3869.489468] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3870.803885] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3873.780097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3873.783180] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3873.808305] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3873.912446] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3875.100477] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3875.162831] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3877.344783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3877.347253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.351378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3877.353261] Lustre: Skipped 2 previous similar messages [ 3890.143074] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3890.173711] Lustre: server umount lustre-OST0000 complete [ 3894.241921] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3894.243169] LustreError: 150657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3894.245509] LustreError: Skipped 2 previous similar messages [ 3894.252935] LustreError: 150657:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3897.492396] Lustre: server umount lustre-OST0001 complete [ 3913.183100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3913.252453] Lustre: server umount lustre-MDT0000 complete [ 3914.505265] LustreError: 155279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432237 with bad export cookie 1916095918648963524 [ 3914.507713] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3914.512380] LustreError: 155279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3914.640794] Lustre: server umount lustre-MDT0001 complete [ 3922.278099] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 20:30:44 (1789432244) [ 3922.829414] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3923.334943] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3923.830173] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 20:30:46 (1789432246) [ 3925.960457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.044654] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3926.054867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3927.305163] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3929.880703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.907479] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3929.918066] Lustre: 157094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3931.190923] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3933.478979] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3935.143836] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3937.178711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.238322] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3938.988504] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3941.096812] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3942.368493] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3942.410478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3943.459821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3944.690427] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3944.757465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3947.488433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3947.490701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3947.496658] Lustre: Skipped 5 previous similar messages [ 3951.435628] Lustre: server umount lustre-OST0000 complete [ 3964.977348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3967.019386] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 3968.163971] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3968.169567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3968.169641] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3968.176049] Lustre: Skipped 1 previous similar message [ 3969.291027] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3970.746257] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3970.811747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3972.117779] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3972.194985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3973.892031] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a148440a800.ost_server_uuid 50 [ 3974.384022] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9a148440a800.ost_server_uuid in FULL state after 0 sec [ 3974.434190] LustreError: 157101:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.5@tcp) returned error from blocking AST (req@00000000eca31960 x1876355238255872 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f8534e9b/0x1a9757160aa5bb02 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.5@tcp remote: 0x97bde928f9e25a94 expref: 5 pid: 157101 timeout: 4074 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3974.444605] LustreError: lustre-MDT0000: A client on nid 192.168.203.5@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3974.447523] LustreError: 157089:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.5@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f8534e9b/0x1a9757160aa5bb02 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.5@tcp remote: 0x97bde928f9e25a94 expref: 6 pid: 157101 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3981.516276] Lustre: server umount lustre-OST0000 complete [ 3988.824137] Lustre: server umount lustre-MDT0000 complete [ 3988.959873] LustreError: 157104:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.966704] LustreError: 157104:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3989.971172] LustreError: 157086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432312 with bad export cookie 1916095918648964693 [ 3989.974208] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3989.974717] LustreError: 157086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4004.319110] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4004.396217] Lustre: server umount lustre-MDT0001 complete [ 4007.567665] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4008.565851] Key type lgssc unregistered [ 4008.701279] LNet: 163203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4008.704041] LNetError: 163203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4008.710245] LNet: Removed LNI 192.168.203.105@tcp [ 4008.994030] Key type .llcrypt unregistered [ 4008.995770] Key type ._llcrypt unregistered [ 4017.453902] Key type ._llcrypt registered [ 4017.455077] Key type .llcrypt registered [ 4017.499615] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4017.831430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4017.873084] alg: No test for adler32 (adler32-zlib) [ 4018.732729] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4018.833416] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4020.424317] Key type lgssc registered [ 4020.803216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4023.438672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4023.446869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4024.596459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.970109] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4028.702617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.833832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4030.185447] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4032.350915] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4033.912282] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4035.950645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4036.049816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4037.752728] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4039.931328] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4049.384423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4052.830326] Lustre: Failing over lustre-MDT0000 [ 4053.086841] Lustre: server umount lustre-MDT0000 complete [ 4054.498776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.499657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4054.499934] LustreError: 164583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4054.499949] LustreError: 164583:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4054.505252] Lustre: Skipped 2 previous similar messages [ 4059.617093] LustreError: 164587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.624282] LustreError: 164587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4063.985119] LustreError: 164612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.5@tcp arrived at 1789432386 with bad export cookie 311752439925277354 [ 4063.990442] LustreError: 164612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4064.736758] LustreError: 167076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.742691] LustreError: 167076:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4066.242574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.277528] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4066.364330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4066.384472] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4067.617078] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4068.308884] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4068.312512] Lustre: lustre-MDT0000: Denying connection for new client a85343ec-5184-4478-b5f6-df002c2c6f46 (at 192.168.203.5@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 4071.396551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4076.513353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4076.515147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4076.518784] Lustre: Skipped 1 previous similar message [ 4081.632194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4081.634211] Lustre: Skipped 1 previous similar message [ 4086.753537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.755461] Lustre: Skipped 1 previous similar message [ 4089.311087] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4089.359318] Lustre: server umount lustre-OST0000 complete [ 4090.498228] LustreError: 168336:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4090.500931] Lustre: 167685:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4090.504336] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4090.510185] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4090.514309] LustreError: 167685:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008b43426c x1876355806433536/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4090.520125] LustreError: 167685:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4090.523144] LustreError: 167685:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4090.525881] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 4090.623372] Lustre: server umount lustre-MDT0000 complete [ 4091.872696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.876906] LustreError: 164584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.878957] LustreError: 164569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432414 with bad export cookie 311752439925277711 [ 4091.881638] LustreError: 163897:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4091.882823] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4091.885044] LustreError: 164569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4098.031900] Lustre: server umount lustre-MDT0001 complete [ 4101.378725] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4102.326623] Key type lgssc unregistered [ 4102.450337] LNet: 169086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4102.453461] LNetError: 169086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4102.465280] LNet: Removed LNI 192.168.203.105@tcp [ 4102.746080] Key type .llcrypt unregistered [ 4102.747555] Key type ._llcrypt unregistered [ 4111.550033] Key type ._llcrypt registered [ 4111.551354] Key type .llcrypt registered [ 4111.589841] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4111.993940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4112.004863] alg: No test for adler32 (adler32-zlib) [ 4112.872557] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4112.968844] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4114.551159] Key type lgssc registered [ 4114.918595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4117.573957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4117.582537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4118.736892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4120.174279] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4123.028203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4123.152620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4124.443384] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4126.823514] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4128.643303] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4130.837714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4130.967072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.825867] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4135.174475] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4144.293688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4148.704309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4148.707786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4148.712779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4149.729791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4149.729878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4149.733051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.104210] Lustre: server umount lustre-OST0000 complete [ 4154.848204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.852066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4154.853792] Lustre: Skipped 2 previous similar messages [ 4159.967888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4159.969963] Lustre: Skipped 1 previous similar message [ 4160.403153] Lustre: server umount lustre-MDT0000 complete [ 4161.564105] LustreError: 170453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432484 with bad export cookie 6268853396042143591 [ 4161.566859] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4161.569145] LustreError: 170453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4161.670526] Lustre: server umount lustre-MDT0001 complete [ 4164.908328] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4165.807688] Key type lgssc unregistered [ 4165.932358] LNet: 174016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4165.934538] LNetError: 174016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4165.947263] LNet: Removed LNI 192.168.203.105@tcp [ 4166.221476] Key type .llcrypt unregistered [ 4166.222474] Key type ._llcrypt unregistered [ 4176.884164] Key type ._llcrypt registered [ 4176.885312] Key type .llcrypt registered [ 4176.921326] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4177.333177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4177.341160] alg: No test for adler32 (adler32-zlib) [ 4178.190022] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4178.274237] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4179.855156] Key type lgssc registered [ 4180.193461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4182.613294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4182.621301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4183.765597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.160355] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4188.051970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4188.189275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4189.624248] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4192.153594] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4194.173301] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4196.582816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4196.714975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4198.728895] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4201.093149] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4210.022237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4214.239751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4214.242420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4214.246285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4215.266062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.266084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4215.266352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4215.274479] Lustre: Skipped 1 previous similar message [ 4218.825389] Lustre: server umount lustre-OST0000 complete [ 4220.384607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4220.385363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4220.388429] Lustre: Skipped 1 previous similar message [ 4225.503935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4225.505592] Lustre: Skipped 2 previous similar messages [ 4226.128668] Lustre: server umount lustre-MDT0000 complete [ 4227.215198] LustreError: 175669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432550 with bad export cookie 1257962202309281286 [ 4227.217547] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4227.218690] LustreError: 175669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4227.321519] Lustre: server umount lustre-MDT0001 complete [ 4230.526380] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4231.477141] Key type lgssc unregistered [ 4231.602327] LNet: 179239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4231.604605] LNetError: 179239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4231.616279] LNet: Removed LNI 192.168.203.105@tcp [ 4231.895623] Key type .llcrypt unregistered [ 4231.897272] Key type ._llcrypt unregistered [ 4239.824266] Key type ._llcrypt registered [ 4239.825210] Key type .llcrypt registered [ 4239.860890] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4240.185838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4240.229748] alg: No test for adler32 (adler32-zlib) [ 4241.097193] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4241.186140] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4242.767157] Key type lgssc registered [ 4243.096508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4246.569800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4246.575805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.707386] Lustre: *** cfs_fail_loc=135, val=0*** [ 4247.746360] LustreError: 180733:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 4247.748617] LustreError: 180733:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.105@tcp: cfg command failed: rc = -2 [ 4247.752103] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4247.752103] [ 4247.757134] LustreError: MGC192.168.203.105@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4247.762666] LustreError: 180695:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4247.766570] LustreError: 180695:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 4247.769264] LustreError: 180695:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 4247.772154] LustreError: 180695:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 4247.804421] Lustre: server umount lustre-MDT0000 complete [ 4247.806500] LustreError: 180695:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4250.093645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.228996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4251.644774] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4254.574865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.706180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4256.191339] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4258.865148] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4260.798488] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4264.608885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.615891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4264.618351] Lustre: Skipped 1 previous similar message [ 4267.672014] Lustre: server umount lustre-MDT0000 complete [ 4268.987318] LustreError: 181087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432591 with bad export cookie 7732750756758084569 [ 4268.990368] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4268.991941] LustreError: 181087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4269.110714] Lustre: server umount lustre-MDT0001 complete [ 4272.880861] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4273.929775] Key type lgssc unregistered [ 4274.068415] LNet: 183423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4274.072645] LNetError: 183423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4274.079313] LNet: Removed LNI 192.168.203.105@tcp [ 4274.395505] Key type .llcrypt unregistered [ 4274.396777] Key type ._llcrypt unregistered [ 4281.278795] Key type ._llcrypt registered [ 4281.280310] Key type .llcrypt registered [ 4281.319576] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4281.426404] Key type .llcrypt unregistered [ 4281.428164] Key type ._llcrypt unregistered [ 4288.460713] Key type ._llcrypt registered [ 4288.462022] Key type .llcrypt registered [ 4288.499644] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4288.817620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4288.852606] alg: No test for adler32 (adler32-zlib) [ 4289.698732] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4289.791141] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4291.375232] Key type lgssc registered [ 4291.760595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4294.390789] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4294.397168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4309.855468] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4314.976799] LustreError: 185521:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.105@tcp: failed processing log, type 4: rc = -110 [ 4341.599281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4343.545629] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4345.954286] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4348.312426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.484976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4348.510111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4349.900032] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4352.744716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.868728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4354.207206] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4356.638178] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4358.554997] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4359.611227] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4363.232218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4363.235552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.241382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4363.744093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4363.747084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.751943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4367.329476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4369.293654] Lustre: server umount lustre-OST0000 complete [ 4372.448804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.449470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4372.454536] Lustre: Skipped 1 previous similar message [ 4372.459476] Lustre: Skipped 2 previous similar messages [ 4376.731409] Lustre: server umount lustre-MDT0000 complete [ 4377.568126] LustreError: 186562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.573794] LustreError: 186562:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4378.047366] LustreError: 185527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432700 with bad export cookie 3928289988624186827 [ 4378.050888] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.051705] LustreError: 185527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4378.178942] Lustre: server umount lustre-MDT0001 complete [ 4381.605687] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4382.614173] Key type lgssc unregistered [ 4382.738256] LNet: 189386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4382.740630] LNetError: 189386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4382.749277] LNet: Removed LNI 192.168.203.105@tcp [ 4383.044198] Key type .llcrypt unregistered [ 4383.045313] Key type ._llcrypt unregistered [ 4391.631614] Key type ._llcrypt registered [ 4391.633222] Key type .llcrypt registered [ 4391.671071] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4391.987416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4391.996042] alg: No test for adler32 (adler32-zlib) [ 4392.851616] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4392.948109] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4394.535153] Key type lgssc registered [ 4394.904310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4397.636471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4397.644708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.811818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4400.294551] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4403.269080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4403.401194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4404.874269] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4407.455363] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4409.255990] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4411.395518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4411.522353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4413.450877] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4415.806292] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4423.849299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4427.711322] Lustre: Failing over lustre-MDT0000 [ 4427.802321] Lustre: server umount lustre-MDT0000 complete [ 4429.283578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4429.283590] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.284746] LustreError: 192186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4429.284771] LustreError: 192186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4429.304088] Lustre: Skipped 1 previous similar message [ 4431.605733] LustreError: 192612:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4431.614243] LustreError: 192612:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4434.400522] LustreError: 190771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.406125] LustreError: 190771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4436.716395] LustreError: 190767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4440.785190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.835897] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4440.928479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.952236] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4441.838870] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4442.374492] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4443.950279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4446.180726] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4446.188914] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4446.207577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4448.478699] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4459.999959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4460.003391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.008103] Lustre: Skipped 1 previous similar message [ 4460.010536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4461.536953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4461.537885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4461.540073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4461.541316] LustreError: Skipped 1 previous similar message [ 4461.544147] Lustre: Skipped 1 previous similar message [ 4462.797694] Lustre: server umount lustre-OST0000 complete [ 4466.657425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4466.657646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4466.658272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4466.670073] Lustre: Skipped 1 previous similar message [ 4470.360844] Lustre: server umount lustre-MDT0000 complete [ 4471.654379] LustreError: 194614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432794 with bad export cookie 4587325233285019868 [ 4471.657517] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4471.660723] LustreError: 194614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4471.770468] Lustre: server umount lustre-MDT0001 complete [ 4475.511037] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4476.519945] Key type lgssc unregistered [ 4476.633303] LNet: 195670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4476.635837] LNetError: 195670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4476.643291] LNet: Removed LNI 192.168.203.105@tcp [ 4476.950448] Key type .llcrypt unregistered [ 4476.951909] Key type ._llcrypt unregistered [ 4485.693526] Key type ._llcrypt registered [ 4485.694893] Key type .llcrypt registered [ 4485.737116] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4486.054259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4486.065776] alg: No test for adler32 (adler32-zlib) [ 4486.927157] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4487.025381] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4488.615211] Key type lgssc registered [ 4489.022220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4491.847278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4491.853474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.997888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.437165] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4497.267251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4497.388810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4498.712682] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4501.144161] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4503.075272] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4505.449909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4505.583717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4507.672510] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4510.242354] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4517.864232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4521.372711] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4526.933698] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4534.441425] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4543.456448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4543.457670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.460117] LustreError: Skipped 1 previous similar message [ 4543.465366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4543.469753] Lustre: Skipped 1 previous similar message [ 4546.710844] Lustre: server umount lustre-OST0000 complete [ 4548.576586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4548.576728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.577283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4548.589186] Lustre: Skipped 2 previous similar messages [ 4553.697431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4553.700455] Lustre: Skipped 2 previous similar messages [ 4554.336849] Lustre: server umount lustre-MDT0000 complete [ 4555.857888] LustreError: 197037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432878 with bad export cookie 1020964225742850748 [ 4555.860676] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4555.863684] LustreError: 197037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4556.025516] Lustre: server umount lustre-MDT0001 complete [ 4560.324147] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4561.550951] Key type lgssc unregistered [ 4561.715497] LNet: 200918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4561.718811] LNetError: 200918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4561.726499] LNet: Removed LNI 192.168.203.105@tcp [ 4562.102939] Key type .llcrypt unregistered [ 4562.104887] Key type ._llcrypt unregistered [ 4570.233838] Key type ._llcrypt registered [ 4570.235435] Key type .llcrypt registered [ 4570.277416] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4570.697625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4570.714721] alg: No test for adler32 (adler32-zlib) [ 4571.572800] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4571.666825] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4573.247196] Key type lgssc registered [ 4573.665847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4576.629541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4576.637696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.830886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4579.442634] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4582.833108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4582.989353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4584.517448] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4587.320511] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4589.355583] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4591.834968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4591.954365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4593.957444] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4596.407354] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4603.240777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4608.323371] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4611.159419] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4617.095613] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4625.561187] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4639.201318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4639.204235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.210952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4639.214963] Lustre: Skipped 1 previous similar message [ 4642.515319] Lustre: server umount lustre-OST0000 complete [ 4644.321330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4644.321611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4644.322823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4644.325084] Lustre: Skipped 2 previous similar messages [ 4644.332977] LustreError: Skipped 1 previous similar message [ 4649.440740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4649.443568] Lustre: Skipped 2 previous similar messages [ 4654.561062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4654.565320] Lustre: Skipped 1 previous similar message [ 4658.143145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4658.212219] Lustre: server umount lustre-MDT0000 complete [ 4659.674425] LustreError: 204851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789432982 with bad export cookie 13352704985858343888 [ 4659.676755] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4659.682684] LustreError: 204851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4659.814503] Lustre: server umount lustre-MDT0001 complete [ 4663.836123] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4665.000130] Key type lgssc unregistered [ 4665.140424] LNet: 207385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4665.143486] LNetError: 207385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4665.154503] LNet: Removed LNI 192.168.203.105@tcp [ 4665.516450] Key type .llcrypt unregistered [ 4665.518555] Key type ._llcrypt unregistered [ 4675.193647] Key type ._llcrypt registered [ 4675.195430] Key type .llcrypt registered [ 4675.236283] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4675.570221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4675.615547] alg: No test for adler32 (adler32-zlib) [ 4676.479084] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4676.578731] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4678.159183] Key type lgssc registered [ 4678.570204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4681.610477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4681.618975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.845630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4684.539573] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4687.983340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.133627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4689.817944] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4692.830742] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4695.049544] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4697.646332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4697.775404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.912624] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4702.561274] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4708.006923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4713.443962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4713.444572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4713.457953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4714.464632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4714.467527] Lustre: Skipped 1 previous similar message [ 4714.708562] Lustre: server umount lustre-OST0000 complete [ 4718.234500] Lustre: server umount lustre-MDT0000 complete [ 4719.072143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4719.075341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.083344] LustreError: 208772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.092045] LustreError: 208772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4719.585792] LustreError: 210008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.659407] LustreError: 208753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789433042 with bad export cookie 17033603252933349175 [ 4719.663523] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4719.666940] LustreError: 208753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4725.885426] Lustre: server umount lustre-MDT0001 complete [ 4729.781619] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4730.876239] Key type lgssc unregistered [ 4731.009689] LNet: 212448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4731.013801] LNetError: 212448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4731.024335] LNet: Removed LNI 192.168.203.105@tcp [ 4731.355768] Key type .llcrypt unregistered [ 4731.357377] Key type ._llcrypt unregistered [ 4739.424809] Key type ._llcrypt registered [ 4739.426266] Key type .llcrypt registered [ 4739.470697] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4739.811281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4739.833070] alg: No test for adler32 (adler32-zlib) [ 4740.710574] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4740.812195] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4742.399205] Key type lgssc registered [ 4742.905955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4746.169505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4746.180762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4747.377538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4748.915788] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4752.179456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4752.348780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4753.911721] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4756.674709] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4758.729861] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4761.254357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4761.380537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4763.463805] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4766.028047] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4772.646770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4783.076457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4783.077304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.079355] LustreError: Skipped 1 previous similar message [ 4783.085053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4783.088654] Lustre: Skipped 1 previous similar message [ 4786.446361] Lustre: server umount lustre-OST0000 complete [ 4788.192813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4788.193131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4788.196654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4788.200073] Lustre: Skipped 2 previous similar messages [ 4788.204837] Lustre: Skipped 1 previous similar message [ 4793.312560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4793.314553] Lustre: Skipped 1 previous similar message [ 4793.816979] Lustre: server umount lustre-MDT0000 complete [ 4795.006220] LustreError: 213669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789433117 with bad export cookie 4106215230791787289 [ 4795.008935] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4795.011021] LustreError: 213669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4795.129911] Lustre: server umount lustre-MDT0001 complete [ 4798.696437] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4799.761495] Key type lgssc unregistered [ 4799.882460] LNet: 217388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4799.886237] LNetError: 217388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4799.893407] LNet: Removed LNI 192.168.203.105@tcp [ 4800.217141] Key type .llcrypt unregistered [ 4800.218858] Key type ._llcrypt unregistered [ 4807.823511] Key type ._llcrypt registered [ 4807.825103] Key type .llcrypt registered [ 4807.866890] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4808.162947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4808.169519] alg: No test for adler32 (adler32-zlib) [ 4809.030525] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4809.126662] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4810.727160] Key type lgssc registered [ 4811.103713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4813.839665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4813.846789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4815.010806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4816.585240] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4819.810935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4819.966562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4821.474716] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4824.128134] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4826.022318] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4828.394656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4828.537704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4830.558148] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4833.191823] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4840.806190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4846.356223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4846.386943] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4846.458187] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4848.405149] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4850.954160] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4851.504185] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4862.095537] Lustre: Permanently deactivating lustre-OST0001 [ 4862.102945] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4862.598627] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4873.016126] Lustre: Permanently reactivating lustre-OST0001 [ 4873.022786] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4873.026024] Lustre: Skipped 2 previous similar messages [ 4880.898422] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4880.905748] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4880.909870] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4880.918455] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4880.940023] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4884.964354] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.966268] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4884.969797] Lustre: Skipped 2 previous similar messages [ 4884.974899] Lustre: Skipped 1 previous similar message [ 4890.083810] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4890.086704] Lustre: Skipped 1 previous similar message [ 4890.682549] Lustre: server umount lustre-OST0001 complete [ 4892.127850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4892.130794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4892.135494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4895.202193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4895.202214] LustreError: 220625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4895.205148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.209571] LustreError: 220625:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4898.066814] Lustre: server umount lustre-OST0000 complete [ 4900.320275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4900.321977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4900.325288] Lustre: Skipped 1 previous similar message [ 4900.327740] Lustre: Skipped 1 previous similar message [ 4905.501213] Lustre: server umount lustre-MDT0000 complete [ 4906.739491] LustreError: 221484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789433229 with bad export cookie 10595764093721114594 [ 4906.742414] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4906.744847] LustreError: 221484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4906.872109] Lustre: server umount lustre-MDT0001 complete [ 4910.669858] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 4911.754892] Key type lgssc unregistered [ 4911.887458] LNet: 223709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4911.890148] LNetError: 223709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4911.898400] LNet: Removed LNI 192.168.203.105@tcp [ 4912.228363] Key type .llcrypt unregistered [ 4912.230131] Key type ._llcrypt unregistered [ 4926.887924] Key type ._llcrypt registered [ 4926.889413] Key type .llcrypt registered [ 4926.930789] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 4927.327931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4927.378672] alg: No test for adler32 (adler32-zlib) [ 4928.251808] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 4928.354162] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 4929.943199] Key type lgssc registered [ 4930.386932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4933.223371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4933.232664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4934.336139] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4934.345820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4934.405801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4935.824462] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4938.837911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4938.873710] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4938.885577] Lustre: 225883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4938.983335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4940.447592] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4943.099729] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4945.035730] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4947.263213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4947.331981] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4947.400101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4949.349983] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 4949.792785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 4951.792640] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4953.374504] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4961.987827] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4968.554181] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4976.138243] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4984.755245] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5005.330585] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5005.333047] Lustre: Skipped 2 previous similar messages [ 5043.277371] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5043.280460] Lustre: Skipped 4 previous similar messages [ 5114.925109] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 5114.927808] Lustre: Skipped 8 previous similar messages [ 5118.944919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5118.945548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5118.947892] LustreError: Skipped 1 previous similar message [ 5118.953463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5118.957888] Lustre: Skipped 1 previous similar message [ 5122.454192] Lustre: server umount lustre-OST0000 complete [ 5124.065862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5124.066996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5124.071062] Lustre: Skipped 2 previous similar messages [ 5129.184434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5129.187628] Lustre: Skipped 2 previous similar messages [ 5134.303890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5134.305808] Lustre: Skipped 1 previous similar message [ 5137.887145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5137.947900] Lustre: server umount lustre-MDT0000 complete [ 5139.102204] LustreError: 225875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789433461 with bad export cookie 12027929815518516641 [ 5139.104686] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5139.106474] LustreError: 225875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5139.235205] Lustre: server umount lustre-MDT0001 complete [ 5142.821378] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5143.899135] Key type lgssc unregistered [ 5144.023326] LNet: 230689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5144.026953] LNetError: 230689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5144.038487] LNet: Removed LNI 192.168.203.105@tcp [ 5144.350376] Key type .llcrypt unregistered [ 5144.352099] Key type ._llcrypt unregistered [ 5152.996599] Key type ._llcrypt registered [ 5152.997827] Key type .llcrypt registered [ 5153.032637] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5153.391045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5153.421122] alg: No test for adler32 (adler32-zlib) [ 5154.272804] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 5154.353304] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5155.927182] Key type lgssc registered [ 5156.259430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5158.718982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5158.724640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5159.884925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5161.215314] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5163.948824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.068653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5165.344865] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5167.718508] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5169.516455] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5171.628571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5171.734026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5173.557781] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_default_debug -1 all [ 5175.913992] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5185.062951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5199.327796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5199.330823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.336627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5200.353222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5200.355770] Lustre: Skipped 2 previous similar messages [ 5203.093816] Lustre: server umount lustre-OST0000 complete [ 5204.959827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5204.963406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.967961] Lustre: Skipped 1 previous similar message [ 5204.970493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5210.520779] Lustre: server umount lustre-MDT0000 complete [ 5211.615824] LustreError: 232093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5211.621866] LustreError: 232093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5211.746282] LustreError: 232054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789433534 with bad export cookie 1247649966571281394 [ 5211.749295] LustreError: MGC192.168.203.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5211.751499] LustreError: 232054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5211.866382] Lustre: server umount lustre-MDT0001 complete [ 5215.400575] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5216.376158] Key type lgssc unregistered [ 5216.503435] LNet: 235785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5216.506053] LNetError: 235785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5216.514555] LNet: Removed LNI 192.168.203.105@tcp [ 5216.809060] Key type .llcrypt unregistered [ 5216.810448] Key type ._llcrypt unregistered [ 5224.836313] Key type ._llcrypt registered [ 5224.837572] Key type .llcrypt registered [ 5224.879083] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5225.312449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5225.321711] alg: No test for adler32 (adler32-zlib) [ 5226.166576] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 5226.247649] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5227.823208] Key type lgssc registered [ 5228.155660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5494.658144] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5496.010226] Key type lgssc unregistered [ 5496.158483] LNet: 237781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5496.166311] LNetError: 237781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5496.181596] LNet: Removed LNI 192.168.203.105@tcp [ 5496.591363] Key type .llcrypt unregistered [ 5496.592952] Key type ._llcrypt unregistered [ 5505.956484] Key type ._llcrypt registered [ 5505.958275] Key type .llcrypt registered [ 5506.002152] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing set_hostid [ 5513.175827] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing load_modules_local [ 5513.520727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5513.572174] alg: No test for adler32 (adler32-zlib) [ 5514.456711] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 5514.553532] LNet: Added LNI 192.168.203.105@tcp [8/256/0/180] [ 5516.135218] Key type lgssc registered [ 5516.551175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5527.796517] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5229 sec ========== 20:57:30 (1789433850) [ 5528.435283] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:57:30 (1789433850) === [ 5529.814718] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:57:32 (1789433852) === [ 5545.090843] Lustre: DEBUG MARKER: oleg305-server.virtnet: executing unload_modules_local [ 5546.228281] Key type lgssc unregistered [ 5546.368300] LNet: 242981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5546.371546] LNetError: 242981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5546.379314] LNet: Removed LNI 192.168.203.105@tcp [ 5546.733955] Key type .llcrypt unregistered [ 5546.735802] Key type ._llcrypt unregistered