[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 458798082 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003284] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007721] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.013040] SELinux: Initializing. [ 0.014056] *** VALIDATE selinux *** [ 0.022832] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026795] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028082] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.031261] *** VALIDATE proc *** [ 0.033000] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034259] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036134] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038024] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042173] debug: unmapping init [mem 0xffffffffa7459000-0xffffffffa7460fff] [ 0.044163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045701] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052024] ... event mask: 000000070000000f [ 0.053299] rcu: Hierarchical SRCU implementation. [ 0.055404] smp: Bringing up secondary CPUs ... [ 0.056558] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060406] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143065] node 0 deferred pages initialised in 78ms [ 0.146429] devtmpfs: initialized [ 0.148234] x86/mm: Memory block size: 128MB [ 0.149000] gcov: version magic: 0x41383552 [ 0.152117] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156265] pinctrl core: initialized pinctrl subsystem [ 0.158174] [ 0.158702] ************************************************************* [ 0.161011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163009] ** ** [ 0.165009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166009] ** ** [ 0.168010] ** This means that this kernel is built to expose internal ** [ 0.170010] ** IOMMU data structures, which may compromise security on ** [ 0.172009] ** your system. ** [ 0.174012] ** ** [ 0.176009] ** If you see this message and you are not debugging the ** [ 0.178011] ** kernel, report this immediately to your vendor! ** [ 0.180008] ** ** [ 0.181009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183009] ************************************************************* [ 0.185585] NET: Registered protocol family 16 [ 0.187398] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194103] cpuidle: using governor menu [ 0.195545] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.197549] PCI: Using configuration type 1 for base access [ 0.200119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.210124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216222] cryptd: max_cpu_qlen set to 1000 [ 0.219356] ACPI: Added _OSI(Module Device) [ 0.221019] ACPI: Added _OSI(Processor Device) [ 0.223021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.229653] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237020] ACPI: Interpreter enabled [ 0.238058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238978] ACPI: Using IOAPIC for interrupt routing [ 0.241115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244727] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254848] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260046] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.263140] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269196] acpiphp: Slot [2] registered [ 0.272318] acpiphp: Slot [5] registered [ 0.273190] acpiphp: Slot [6] registered [ 0.275144] acpiphp: Slot [7] registered [ 0.277180] acpiphp: Slot [8] registered [ 0.278240] acpiphp: Slot [9] registered [ 0.280149] acpiphp: Slot [10] registered [ 0.281156] acpiphp: Slot [3] registered [ 0.283163] acpiphp: Slot [4] registered [ 0.284099] acpiphp: Slot [11] registered [ 0.285109] acpiphp: Slot [12] registered [ 0.287139] acpiphp: Slot [13] registered [ 0.288114] acpiphp: Slot [14] registered [ 0.290128] acpiphp: Slot [15] registered [ 0.292119] acpiphp: Slot [16] registered [ 0.294127] acpiphp: Slot [17] registered [ 0.295085] acpiphp: Slot [18] registered [ 0.297087] acpiphp: Slot [19] registered [ 0.298084] acpiphp: Slot [20] registered [ 0.299100] acpiphp: Slot [21] registered [ 0.301119] acpiphp: Slot [22] registered [ 0.302182] acpiphp: Slot [23] registered [ 0.304102] acpiphp: Slot [24] registered [ 0.305103] acpiphp: Slot [25] registered [ 0.307117] acpiphp: Slot [26] registered [ 0.308111] acpiphp: Slot [27] registered [ 0.310126] acpiphp: Slot [28] registered [ 0.312128] acpiphp: Slot [29] registered [ 0.313075] acpiphp: Slot [30] registered [ 0.314105] acpiphp: Slot [31] registered [ 0.316057] PCI host bridge to bus 0000:00 [ 0.317018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.320026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.322027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.325039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.330036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.333876] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.336219] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347027] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357033] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359042] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365590] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.369848] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.373043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.376782] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.383018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.404013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.410522] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.454024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.467510] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.477018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.484017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.509016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.522136] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.528020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.536020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.558021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.568550] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.575018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.582016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.598017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.614629] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.621019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.628019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.649022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.659176] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.666015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.672000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.687019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.697521] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.700606] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.702474] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.706490] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.709299] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.716149] iommu: Default domain type: Passthrough [ 0.718456] SCSI subsystem initialized [ 0.720161] ACPI: bus type USB registered [ 0.722123] usbcore: registered new interface driver usbfs [ 0.724107] usbcore: registered new interface driver hub [ 0.726077] usbcore: registered new device driver usb [ 0.728164] pps_core: LinuxPPS API ver. 1 registered [ 0.730013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.733053] PTP clock support registered [ 0.735144] EDAC MC: Ver: 3.0.0 [ 0.736613] PCI: Using ACPI for IRQ routing [ 0.737868] NetLabel: Initializing [ 0.738012] NetLabel: domain hash size = 128 [ 0.739011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.740105] NetLabel: unlabeled traffic allowed by default [ 0.742295] vgaarb: loaded [ 0.743278] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.746014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.753000] clocksource: Switched to clocksource kvm-clock [ 0.866176] VFS: Disk quotas dquot_6.6.0 [ 0.869355] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.875360] *** VALIDATE ramfs *** [ 0.878295] *** VALIDATE hugetlbfs *** [ 0.883130] pnp: PnP ACPI init [ 0.889766] pnp: PnP ACPI: found 6 devices [ 0.907439] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.910516] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.912839] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.915382] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.918489] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.921585] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.924120] NET: Registered protocol family 2 [ 0.926776] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.932255] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.936954] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.942697] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.947114] TCP: Hash tables configured (established 65536 bind 65536) [ 0.950382] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.953649] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.956900] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960131] NET: Registered protocol family 1 [ 0.962947] RPC: Registered named UNIX socket transport module. [ 0.965384] RPC: Registered udp transport module. [ 0.967257] RPC: Registered tcp transport module. [ 0.969200] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.971942] NET: Registered protocol family 44 [ 0.973936] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.976503] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.978931] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.981643] PCI: CLS 0 bytes, default 64 [ 0.983576] Unpacking initramfs... [ 2.415355] debug: unmapping init [mem 0xffff88887cc54000-0xffff88887ffbffff] [ 2.419505] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.421764] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.425098] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.926523] Initialise system trusted keyrings [ 2.928643] Key type blacklist registered [ 2.931130] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.941773] zbud: loaded [ 2.945619] *** VALIDATE nfs *** [ 2.947137] *** VALIDATE nfs4 *** [ 2.949381] pstore: using deflate compression [ 2.953115] Platform Keyring initialized [ 3.063792] NET: Registered protocol family 38 [ 3.065874] Key type asymmetric registered [ 3.067686] Asymmetric key parser 'x509' registered [ 3.069679] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.073210] io scheduler mq-deadline registered [ 3.075342] io scheduler kyber registered [ 3.077269] io scheduler bfq registered [ 3.079506] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.083315] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.086695] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.090405] ACPI: Power Button [PWRF] [ 3.095870] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.103332] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.117160] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.123052] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.135601] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.162636] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.191211] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.195337] Non-volatile memory driver v1.3 [ 3.196561] Linux agpgart interface v0.103 [ 3.225720] virtio_blk virtio1: [vda] 146944 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.228056] vda: detected capacity change from 0 to 75235328 [ 3.243788] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.246920] vdb: detected capacity change from 0 to 1073741824 [ 3.267462] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.270868] vdc: detected capacity change from 0 to 2621440000 [ 3.288596] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.292229] vdd: detected capacity change from 0 to 2621440000 [ 3.306083] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.308436] vde: detected capacity change from 0 to 4294967296 [ 3.322909] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.326712] vdf: detected capacity change from 0 to 4294967296 [ 3.332853] libphy: Fixed MDIO Bus: probed [ 3.338556] usbcore: registered new interface driver usbserial_generic [ 3.341491] usbserial: USB Serial support registered for generic [ 3.343802] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.348588] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.350585] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.353416] mousedev: PS/2 mouse device common for all mice [ 3.357117] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.358471] rtc_cmos 00:05: RTC can wake from S4 [ 3.365706] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.366219] rtc_cmos 00:05: registered as rtc0 [ 3.370656] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.370870] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.374239] intel_pstate: CPU model not supported [ 3.377959] hid: raw HID events driver (C) Jiri Kosina [ 3.384301] usbcore: registered new interface driver usbhid [ 3.386870] usbhid: USB HID core driver [ 3.388519] drop_monitor: Initializing network drop monitor service [ 3.390466] Initializing XFRM netlink socket [ 3.392153] NET: Registered protocol family 10 [ 3.394859] Segment Routing with IPv6 [ 3.396594] NET: Registered protocol family 17 [ 3.399334] mpls_gso: MPLS GSO support [ 3.406415] RAS: Correctable Errors collector initialized. [ 3.408687] AVX version of gcm_enc/dec engaged. [ 3.410463] AES CTR mode by8 optimization enabled [ 3.493948] sched_clock: Marking stable (3493911222, 0)->(4433993041, -940081819) [ 3.497710] registered taskstats version 1 [ 3.500316] Loading compiled-in X.509 certificates [ 3.502565] zswap: loaded using pool lzo/zbud [ 3.527061] Key type big_key registered [ 3.543082] Key type encrypted registered [ 3.544749] ima: No TPM chip found, activating TPM-bypass! [ 3.547029] ima: Allocated hash algorithm: sha1 [ 3.548759] ima: No architecture policies found [ 3.550528] evm: Initialising EVM extended attributes: [ 3.552081] evm: security.selinux [ 3.553056] evm: security.ima [ 3.554244] evm: security.capability [ 3.555702] evm: HMAC attrs: 0x1 [ 3.557980] rtc_cmos 00:05: setting system clock to 2026-08-27 14:11:25 UTC (1787839885) [ 3.564552] debug: unmapping init [mem 0xffffffffa8403000-0xffffffffa85fffff] [ 3.568154] debug: unmapping init [mem 0xffffffffa7182000-0xffffffffa7458fff] [ 3.579152] Write protecting the kernel read-only data: 28672k [ 3.582642] debug: unmapping init [mem 0xffffffffa5803000-0xffffffffa59fffff] [ 3.585778] debug: unmapping init [mem 0xffffffffa6114000-0xffffffffa61fffff] [ 3.620757] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.628235] systemd[1]: Detected virtualization kvm. [ 3.629955] systemd[1]: Detected architecture x86-64. [ 3.631887] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.660348] systemd[1]: No hostname configured. [ 3.662258] systemd[1]: Set hostname to . [ 3.664223] random: systemd: uninitialized urandom read (16 bytes read) [ 3.666380] systemd[1]: Initializing machine ID from random generator. [ 3.715519] random: ln: uninitialized urandom read (6 bytes read) [ 3.814877] random: systemd: uninitialized urandom read (16 bytes read) [ 3.818187] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.824856] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.836201] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.534525] device-mapper: uevent: version 1.0.3 [ 4.537116] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.277620] random: fast init done [ 5.280831] virtio_net virtio0 ens2: renamed from eth0 [ 5.364982] scsi host0: ata_piix [ 5.374477] scsi host1: ata_piix [ 5.376088] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.377872] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.045697] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.061033] random: crng init done [ 10.062903] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.524946] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.707923] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.957515] SELinux: Disabled at runtime. [ 12.022146] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.031464] systemd[1]: Detected virtualization kvm. [ 12.033315] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.529153] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.532764] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.538329] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.543337] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.547084] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.557042] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.562274] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on[ 12.751036] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.093280] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.457272] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.479435] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.650252] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.666122] EDAC sbridge: Ver: 1.1.2 [ 15.483870] Key type dns_resolver registered [ 15.796105] NFS: Registering the id_resolver key type [ 15.798445] Key type id_resolver registered [ 15.800158] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg328-server login: [ 52.083016] hrtimer: interrupt took 6731883 ns [ 57.957944] libcfs: loading out-of-tree module taints kernel. [ 58.045501] Key type ._llcrypt registered [ 58.049603] Key type .llcrypt registered [ 58.167481] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 74.872105] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 76.452241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.472411] alg: No test for adler32 (adler32-zlib) [ 77.989939] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 78.976167] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 80.775228] Key type lgssc registered [ 82.709664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.753296] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.129448] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 154.028451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.066125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.436380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.485140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.570685] Lustre: lustre-MDT0000: new disk, initializing [ 155.703940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.743884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.070597] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.543670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.653830] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.682969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.687344] Lustre: Skipped 1 previous similar message [ 173.832968] Lustre: lustre-MDT0001: new disk, initializing [ 173.938549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.991976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 174.010330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 179.016719] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.592913] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 192.948985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.181644] Lustre: lustre-OST0000: new disk, initializing [ 193.185448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.190468] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 193.281920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.704626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 198.723340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.840554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.291043] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.339245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.503292] Lustre: lustre-OST0001: new disk, initializing [ 213.516083] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 213.533405] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.611302] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.680751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.692336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.793638] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.549673] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.727273] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.889371] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.041698] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing check_logdir /tmp/testlogs/ [ 253.082580] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing yml_node [ 259.215936] Lustre: DEBUG MARKER: Client: 2.17.57.95 [ 261.925987] Lustre: DEBUG MARKER: MDS: 2.17.57.95 [ 264.308350] Lustre: DEBUG MARKER: OSS: 2.17.57.95 [ 266.023742] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:15:46 EDT 2026 [ 282.430679] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 283.849410] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.536551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.537670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.555796] Lustre: Skipped 3 previous similar messages [ 301.562113] Lustre: Skipped 2 previous similar messages [ 305.482097] Lustre: server umount lustre-MDT0000 complete [ 309.250971] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840191 with bad export cookie 2344408520550800196 [ 309.253394] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.268378] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 311.781395] LustreError: 6514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.785144] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.789227] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.789235] Lustre: Skipped 1 previous similar message [ 311.817552] LustreError: 6514:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 311.836578] Lustre: Skipped 1 previous similar message [ 314.784067] LustreError: 10506:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.784583] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 314.804086] LustreError: 10506:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 314.829138] Lustre: Skipped 2 previous similar messages [ 315.751744] Lustre: server umount lustre-MDT0001 complete [ 319.955580] Lustre: server umount lustre-OST0000 complete [ 324.108358] Lustre: server umount lustre-OST0001 complete [ 332.306333] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 342.955510] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 381.806415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 382.172581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 382.207069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 382.356609] Lustre: lustre-MDT0000: new disk, initializing [ 382.483493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.528853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 387.642607] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 398.508240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.668226] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 398.741980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 398.750793] Lustre: Skipped 1 previous similar message [ 398.890741] Lustre: lustre-MDT0001: new disk, initializing [ 398.978136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 399.028422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 399.044791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 404.499971] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 414.215253] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 419.978849] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 427.001643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 427.327551] Lustre: lustre-OST0000: new disk, initializing [ 427.331741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 427.337530] Lustre: 17039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 427.442178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.887420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 428.901180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 429.069775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 433.768406] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 444.834793] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 450.515414] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 450.847429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 455.385586] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 455.599966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 459.747873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 459.760939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.781060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 459.796923] Lustre: Skipped 1 previous similar message [ 463.747249] Lustre: server umount lustre-OST0000 complete [ 469.985501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.999728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 470.005787] Lustre: Skipped 2 previous similar messages [ 470.028596] Lustre: Skipped 1 previous similar message [ 474.466776] Lustre: server umount lustre-MDT0000 complete [ 475.110147] LustreError: 15076:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.131961] LustreError: 15076:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 478.418595] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840360 with bad export cookie 2344408520550801729 [ 478.420721] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 478.437637] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 479.088085] Lustre: server umount lustre-MDT0001 complete [ 483.247404] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 10:19:23 (1787840363) [ 491.080091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.566275] LustreError: 19303:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.670483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.127745] LustreError: 19304:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.275470] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 506.346390] LustreError: 19303:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.376187] LustreError: 19303:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 506.637601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.585623] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 517.794201] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 523.237661] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 529.939626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.366868] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 544.584839] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 557.031502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.036651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 557.059206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 557.065163] LustreError: Skipped 1 previous similar message [ 557.073575] Lustre: Skipped 1 previous similar message [ 558.407963] Lustre: server umount lustre-OST0000 complete [ 563.682319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 567.266689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.281067] Lustre: Skipped 2 previous similar messages [ 568.590994] Lustre: server umount lustre-MDT0000 complete [ 572.384243] LustreError: 20032:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.417991] LustreError: 20032:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 572.936407] LustreError: 19285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840454 with bad export cookie 2344408520550802611 [ 572.944325] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.946635] LustreError: 19285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 573.302681] Lustre: server umount lustre-MDT0001 complete [ 587.551830] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 592.161656] Key type lgssc unregistered [ 592.545625] LNet: 22843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.555068] LNetError: 22843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.584979] LNet: Removed LNI 192.168.203.128@tcp [ 593.936153] Key type .llcrypt unregistered [ 593.944889] Key type ._llcrypt unregistered [ 620.862518] Key type ._llcrypt registered [ 620.868661] Key type .llcrypt registered [ 621.019176] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 621.943152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 622.041340] alg: No test for adler32 (adler32-zlib) [ 623.086244] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 623.345063] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 625.040768] Key type lgssc registered [ 626.352945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 634.383734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 634.407269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.861748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 639.981124] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 645.095975] LustreError: 24258:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.123472] LustreError: 24258:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 646.854408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.242447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 651.629367] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 658.691441] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 663.448955] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 670.328636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 670.774516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 676.662855] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 681.014871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 683.191565] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 698.337752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 698.350444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.369840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 699.208869] Lustre: server umount lustre-OST0000 complete [ 703.457311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 703.464373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.474438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.531291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.532747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.558225] Lustre: Skipped 1 previous similar message [ 711.649816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 711.665314] Lustre: Skipped 1 previous similar message [ 713.873996] Lustre: server umount lustre-MDT0000 complete [ 716.768431] LustreError: 24253:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.785628] LustreError: 24253:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 718.318459] LustreError: 24237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840600 with bad export cookie 8741529704174562701 [ 718.322108] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 718.341740] LustreError: 24237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 718.844878] Lustre: server umount lustre-MDT0001 complete [ 731.323168] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 734.458395] Key type lgssc unregistered [ 734.718674] LNet: 28234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.728572] LNetError: 28234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.748877] LNet: Removed LNI 192.168.203.128@tcp [ 735.646382] Key type .llcrypt unregistered [ 735.648674] Key type ._llcrypt unregistered [ 758.704310] Key type ._llcrypt registered [ 758.709195] Key type .llcrypt registered [ 758.803879] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 760.057077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 760.110128] alg: No test for adler32 (adler32-zlib) [ 761.241460] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 761.543724] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 763.215352] Key type lgssc registered [ 764.214235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 771.490275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 771.516586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.880501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 776.921264] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 782.316191] LustreError: 29651:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.332905] LustreError: 29651:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 786.488442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.859035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 790.939598] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 800.446838] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 808.732084] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 821.729756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 822.033359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 827.519752] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 834.046505] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 842.551290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 851.944087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 851.954586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.983425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 852.963461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 852.967405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 852.977430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.440318] Lustre: server umount lustre-OST0000 complete [ 863.201855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.207268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 863.219402] Lustre: Skipped 1 previous similar message [ 863.235027] Lustre: Skipped 2 previous similar messages [ 865.364938] Lustre: server umount lustre-MDT0000 complete [ 868.323318] LustreError: 29647:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.334885] LustreError: 29647:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 869.705749] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840751 with bad export cookie 4674559734726411553 [ 869.728347] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 869.730063] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 870.082356] Lustre: server umount lustre-MDT0001 complete [ 879.575955] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 881.792107] Key type lgssc unregistered [ 882.025365] LNet: 33552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.032945] LNetError: 33552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.057910] LNet: Removed LNI 192.168.203.128@tcp [ 882.912788] Key type .llcrypt unregistered [ 882.917679] Key type ._llcrypt unregistered [ 906.914498] Key type ._llcrypt registered [ 906.916975] Key type .llcrypt registered [ 907.103662] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 908.007444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 908.123948] alg: No test for adler32 (adler32-zlib) [ 909.125619] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 909.438644] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 911.224508] Key type lgssc registered [ 912.662920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 922.649821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 922.681984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.238310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.134404] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 934.369626] LustreError: 34967:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.380399] LustreError: 34967:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 939.491882] LustreError: 34966:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 939.733245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.174565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 944.285409] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 952.642655] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 958.854236] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 965.749905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 966.077593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.149058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 972.779308] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 981.408758] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 994.784320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 994.784574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.797383] LustreError: Skipped 1 previous similar message [ 994.803132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.810629] Lustre: Skipped 1 previous similar message [ 999.912428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.918035] Lustre: Skipped 2 previous similar messages [ 1000.864263] Lustre: server umount lustre-OST0000 complete [ 1005.037750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.066686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1005.070360] Lustre: Skipped 1 previous similar message [ 1005.101315] Lustre: Skipped 1 previous similar message [ 1010.145730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1010.850748] Lustre: server umount lustre-MDT0000 complete [ 1015.043607] LustreError: 34948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840896 with bad export cookie 9990234061554456049 [ 1015.057365] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1015.070909] LustreError: 34948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1015.773607] Lustre: server umount lustre-MDT0001 complete [ 1028.953203] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1032.195443] Key type lgssc unregistered [ 1032.535646] LNet: 38525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.551544] LNetError: 38525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.586439] LNet: Removed LNI 192.168.203.128@tcp [ 1033.706875] Key type .llcrypt unregistered [ 1033.708448] Key type ._llcrypt unregistered [ 1058.354297] Key type ._llcrypt registered [ 1058.357523] Key type .llcrypt registered [ 1058.494773] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1059.476647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.634406] alg: No test for adler32 (adler32-zlib) [ 1060.833778] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1061.053021] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1062.771745] Key type lgssc registered [ 1063.902471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1072.898870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1072.915809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.475995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.235976] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1084.897908] LustreError: 39940:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.914058] LustreError: 39940:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1087.612744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.171845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1092.730869] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1100.428343] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1106.483774] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1113.217348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1113.670732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1119.802268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1120.061500] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1127.039684] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1134.709837] Lustre: lustre-OST0000: Not available for connect from 192.168.203.28@tcp (stopping) [ 1135.085621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1135.091617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.104609] Lustre: Skipped 1 previous similar message [ 1139.702244] Lustre: server umount lustre-OST0000 complete [ 1160.161322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1160.169317] LustreError: Skipped 1 previous similar message [ 1160.174152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.191737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1160.198152] Lustre: Skipped 2 previous similar messages [ 1165.792472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.807141] Lustre: Skipped 2 previous similar messages [ 1166.620834] Lustre: server umount lustre-MDT0000 complete [ 1170.091247] LustreError: 39919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841052 with bad export cookie 2414510190333594982 [ 1170.103097] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.112675] LustreError: 39919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1176.595558] Lustre: server umount lustre-MDT0001 complete [ 1187.387562] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1190.525693] Key type lgssc unregistered [ 1190.872647] LNet: 43648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.905113] LNetError: 43648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.928226] LNet: Removed LNI 192.168.203.128@tcp [ 1192.031153] Key type .llcrypt unregistered [ 1192.033357] Key type ._llcrypt unregistered [ 1215.381741] Key type ._llcrypt registered [ 1215.384195] Key type .llcrypt registered [ 1215.501334] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1217.132676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1217.168622] alg: No test for adler32 (adler32-zlib) [ 1218.447367] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1218.689082] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1220.455302] Key type lgssc registered [ 1221.974326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1230.938984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1230.959127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.460119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1237.405851] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1242.600056] LustreError: 45070:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.622188] LustreError: 45070:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1246.302740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.758791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1251.725382] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1259.188326] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1263.593491] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1270.887858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1271.240949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.431535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1278.071708] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1285.221545] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1291.864051] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.28@tcp (stopping) [ 1292.749402] Lustre: server umount lustre-MDT0000 complete [ 1292.778314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1292.781964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.784984] LustreError: 45066:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.810371] Lustre: Skipped 2 previous similar messages [ 1295.941577] LustreError: 45050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841177 with bad export cookie 16734541909341831227 [ 1295.956920] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1295.958863] LustreError: 45050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1296.967125] LustreError: 45064:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.980527] LustreError: 45064:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1296.995792] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.28@tcp (stopping) [ 1297.891902] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.903261] LustreError: 45064:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.903283] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1300.921928] LustreError: 45064:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1300.949398] Lustre: Skipped 1 previous similar message [ 1302.468960] Lustre: server umount lustre-MDT0001 complete [ 1323.874590] Lustre: server umount lustre-OST0000 complete [ 1339.855685] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1343.258489] Key type lgssc unregistered [ 1343.635563] LNet: 48934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.649934] LNetError: 48934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.678644] LNet: Removed LNI 192.168.203.128@tcp [ 1344.606231] Key type .llcrypt unregistered [ 1344.608854] Key type ._llcrypt unregistered [ 1370.646950] Key type ._llcrypt registered [ 1370.650172] Key type .llcrypt registered [ 1370.762359] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1372.007444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1372.059883] alg: No test for adler32 (adler32-zlib) [ 1373.215538] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1373.485434] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1375.151250] Key type lgssc registered [ 1376.344726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1384.226243] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1384.238437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.711572] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1404.833838] LustreError: 50322:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 1431.521113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1437.431231] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1444.613484] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1609.604730] Lustre: server umount lustre-OST0000 complete [ 1625.463945] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1628.368348] Key type lgssc unregistered [ 1628.683756] LNet: 51997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.692175] LNetError: 51997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.730612] LNet: Removed LNI 192.168.203.128@tcp [ 1629.560527] Key type .llcrypt unregistered [ 1629.564720] Key type ._llcrypt unregistered [ 1652.512977] Key type ._llcrypt registered [ 1652.517746] Key type .llcrypt registered [ 1652.654335] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1653.796237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1653.973040] alg: No test for adler32 (adler32-zlib) [ 1655.077138] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1655.303573] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1657.015612] Key type lgssc registered [ 1658.069743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1664.956442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1664.976853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1666.457391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.268914] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1676.778839] LustreError: 53411:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1676.792883] LustreError: 53411:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1677.719949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.068651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1681.802948] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1688.394419] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1693.415645] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1699.850685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1700.295123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.313588] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1711.671391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1713.664656] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1716.714058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1716.715434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1716.732355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1718.135311] Lustre: server umount lustre-OST0000 complete [ 1721.826481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.836968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1721.848519] Lustre: Skipped 2 previous similar messages [ 1721.862049] Lustre: Skipped 1 previous similar message [ 1726.947685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1726.957565] Lustre: Skipped 1 previous similar message [ 1727.778055] Lustre: server umount lustre-MDT0000 complete [ 1731.204317] LustreError: 53392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841613 with bad export cookie 14252533724738265505 [ 1731.216544] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1731.223054] LustreError: 53392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1731.645479] Lustre: server umount lustre-MDT0001 complete [ 1742.869649] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1745.537031] Key type lgssc unregistered [ 1745.816472] LNet: 56943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.839097] LNetError: 56943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.871477] LNet: Removed LNI 192.168.203.128@tcp [ 1746.639544] Key type .llcrypt unregistered [ 1746.643371] Key type ._llcrypt unregistered [ 1768.715760] Key type ._llcrypt registered [ 1768.718504] Key type .llcrypt registered [ 1768.822296] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1769.910544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1769.981316] alg: No test for adler32 (adler32-zlib) [ 1771.074697] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1771.400356] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1773.127530] Key type lgssc registered [ 1774.069942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1780.670527] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1780.698361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1796.319491] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1801.440321] LustreError: 58332:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 1828.127277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.976339] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1843.201382] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1849.867559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.442394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.514836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1854.956495] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1855.469843] LustreError: 58357:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1855.503811] LustreError: 58357:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1860.591301] LustreError: 58358:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1862.645392] LustreError: 58357:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1862.673489] LustreError: 58357:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1864.004270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.476818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1869.639363] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 1879.290472] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1885.499645] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1890.279724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.280056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.313750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1890.321042] Lustre: Skipped 1 previous similar message [ 1893.909340] Lustre: server umount lustre-OST0000 complete [ 1920.992529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1920.998090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.003127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.013984] Lustre: Skipped 2 previous similar messages [ 1921.029697] Lustre: Skipped 1 previous similar message [ 1925.960959] Lustre: server umount lustre-MDT0000 complete [ 1926.131387] LustreError: 59369:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.158783] LustreError: 59369:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1930.073895] LustreError: 58338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841812 with bad export cookie 16424707773083178228 [ 1930.088881] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1930.091170] LustreError: 58338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1936.606782] Lustre: server umount lustre-MDT0001 complete [ 1946.715155] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 1949.594813] Key type lgssc unregistered [ 1949.977722] LNet: 62081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.997930] LNetError: 62081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.015576] LNet: Removed LNI 192.168.203.128@tcp [ 1952.317418] Key type .llcrypt unregistered [ 1952.319581] Key type ._llcrypt unregistered [ 1977.048406] Key type ._llcrypt registered [ 1977.052625] Key type .llcrypt registered [ 1977.227238] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 1978.526184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1978.616172] alg: No test for adler32 (adler32-zlib) [ 1979.727621] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 1980.004191] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 1981.703623] Key type lgssc registered [ 1982.962590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1992.853394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1992.911364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1994.697956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2000.044216] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2004.961557] LustreError: 63495:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.996066] LustreError: 63495:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2010.086555] LustreError: 63494:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.451276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.958898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2016.696812] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2025.453470] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2031.662603] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2040.384838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2041.360896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.524136] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2061.962898] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2064.889932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2080.231779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.233114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2080.233292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2080.247597] Lustre: Skipped 1 previous similar message [ 2085.352532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2085.362023] Lustre: Skipped 2 previous similar messages [ 2085.801905] Lustre: server umount lustre-OST0000 complete [ 2090.471230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.476561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2090.491001] Lustre: Skipped 1 previous similar message [ 2090.511208] Lustre: Skipped 1 previous similar message [ 2095.586579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2095.592907] Lustre: Skipped 1 previous similar message [ 2095.990389] Lustre: server umount lustre-MDT0000 complete [ 2100.102295] LustreError: 63475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841982 with bad export cookie 1747277229443527011 [ 2100.105280] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2100.109145] LustreError: 63475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2100.435761] Lustre: server umount lustre-MDT0001 complete [ 2110.785134] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2113.412867] Key type lgssc unregistered [ 2113.683278] LNet: 67059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.691886] LNetError: 67059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.703719] LNet: Removed LNI 192.168.203.128@tcp [ 2114.480407] Key type .llcrypt unregistered [ 2114.486444] Key type ._llcrypt unregistered [ 2139.128058] Key type ._llcrypt registered [ 2139.130623] Key type .llcrypt registered [ 2139.276367] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2140.672656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2140.697541] alg: No test for adler32 (adler32-zlib) [ 2141.805829] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2142.073852] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2143.799170] Key type lgssc registered [ 2145.079282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2155.242908] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:47:15 (1787842035) [ 2162.395480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2162.421282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.014331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2168.835939] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2173.415481] LustreError: 68762:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.443927] LustreError: 68762:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2177.580321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.923653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2182.525385] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2191.886814] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2199.945647] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2208.225054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2208.642784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.665280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2215.683102] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2223.812838] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2229.658226] Lustre: Failing over lustre-MDT0000 [ 2230.045781] Lustre: server umount lustre-MDT0000 complete [ 2230.944939] LustreError: 68758:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.324436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.351855] LustreError: 68759:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.356326] Lustre: Skipped 2 previous similar messages [ 2233.373283] LustreError: 68759:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2235.980579] LustreError: 68759:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.999439] LustreError: 68759:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2240.509935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.645259] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.790715] Lustre: *** cfs_fail_loc=135, val=0*** [ 2240.892582] LustreError: 71850:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2240.899974] LustreError: 71850:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -2 [ 2240.923225] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2240.923225] [ 2240.933769] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2240.954470] LustreError: 71837:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2240.960636] LustreError: 71837:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2240.966055] LustreError: 71837:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2240.970411] LustreError: 71837:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2240.989794] Lustre: server umount lustre-MDT0000 complete [ 2240.992727] LustreError: 71837:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2241.097401] LustreError: 68758:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.118721] LustreError: 68758:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2248.153900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.335915] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2248.348319] LustreError: 68754:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2248.352909] LustreError: 72195:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.128@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000c5b19947) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2248.376739] LustreError: 67828:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000011dabd8b x1874688238191360/t0(0) o250->MGC192.168.203.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2248.719817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.779242] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2251.349883] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2253.664533] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2253.812143] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2253.851581] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2253.890551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2264.037182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2264.041470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.048807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2264.070594] Lustre: Skipped 1 previous similar message [ 2269.155753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2269.167061] Lustre: Skipped 2 previous similar messages [ 2274.279109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2274.293323] Lustre: Skipped 1 previous similar message [ 2276.319163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2276.558334] Lustre: server umount lustre-OST0000 complete [ 2284.512378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.521454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2284.526739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.534887] Lustre: Skipped 1 previous similar message [ 2286.750597] Lustre: server umount lustre-MDT0000 complete [ 2289.641503] LustreError: 68757:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.670104] LustreError: 68757:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 2290.289316] LustreError: 68742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842172 with bad export cookie 6780546061090009112 [ 2290.306827] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2290.745701] Lustre: server umount lustre-MDT0001 complete [ 2302.396089] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2305.719472] Key type lgssc unregistered [ 2306.054789] LNet: 73633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.074472] LNetError: 73633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.103209] LNet: Removed LNI 192.168.203.128@tcp [ 2306.930708] Key type .llcrypt unregistered [ 2306.941330] Key type ._llcrypt unregistered [ 2331.469321] Key type ._llcrypt registered [ 2331.474303] Key type .llcrypt registered [ 2331.577968] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2332.631559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2332.657650] alg: No test for adler32 (adler32-zlib) [ 2333.892342] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2334.258989] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2335.927215] Key type lgssc registered [ 2337.254541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2345.030449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2345.053713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.558629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.972172] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2356.711623] LustreError: 75047:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.738679] LustreError: 75047:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2359.052860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.545845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2364.025128] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2371.513264] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2376.705702] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2383.247132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2383.521615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.771905] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2392.759234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2397.013205] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2402.566597] Lustre: Failing over lustre-MDT0000 [ 2402.889620] Lustre: server umount lustre-MDT0000 complete [ 2403.301106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.312665] LustreError: 76289:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.332238] Lustre: Skipped 2 previous similar messages [ 2403.366350] LustreError: 76289:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2408.432276] LustreError: 75043:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.476075] LustreError: 75043:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2411.968959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.084775] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2412.200987] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2412.332235] LustreError: 78134:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2412.336721] LustreError: 78134:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -9 [ 2412.345362] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2412.345362] [ 2412.355935] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2412.369739] LustreError: 78122:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2412.377721] LustreError: 78122:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2412.388867] LustreError: 78122:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2412.399816] LustreError: 78122:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2412.425705] Lustre: server umount lustre-MDT0000 complete [ 2412.429909] LustreError: 78122:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2413.538671] LustreError: 75042:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.568909] LustreError: 75042:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2418.662153] LustreError: 76289:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.691132] LustreError: 76289:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2418.874612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.988491] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2419.293812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2419.374677] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2423.856894] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2424.291954] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2424.303553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2424.435084] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2424.487876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2430.948790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.962721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.971021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2434.533481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2434.535860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.656123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2439.660988] Lustre: Skipped 1 previous similar message [ 2444.775796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.788076] Lustre: Skipped 2 previous similar messages [ 2445.281807] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2445.490278] Lustre: server umount lustre-OST0000 complete [ 2449.890310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.917082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2449.929333] Lustre: Skipped 1 previous similar message [ 2455.783559] Lustre: server umount lustre-MDT0000 complete [ 2459.053380] LustreError: 75028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842340 with bad export cookie 16205512264661516696 [ 2459.061786] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2459.070481] LustreError: 75028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2459.486471] Lustre: server umount lustre-MDT0001 complete [ 2469.799097] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2472.762347] Key type lgssc unregistered [ 2473.099994] LNet: 79921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.108322] LNetError: 79921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2473.128289] LNet: Removed LNI 192.168.203.128@tcp [ 2474.036431] Key type .llcrypt unregistered [ 2474.040833] Key type ._llcrypt unregistered [ 2486.276224] Key type ._llcrypt registered [ 2486.279134] Key type .llcrypt registered [ 2486.381024] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2486.656436] Key type .llcrypt unregistered [ 2486.664149] Key type ._llcrypt unregistered [ 2508.402705] Key type ._llcrypt registered [ 2508.404590] Key type .llcrypt registered [ 2508.509139] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2509.845005] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2510.073459] alg: No test for adler32 (adler32-zlib) [ 2511.142233] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2511.408735] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2513.047168] Key type lgssc registered [ 2514.317480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2522.941701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2522.953373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.604761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.931503] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2534.889400] LustreError: 81827:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.917358] LustreError: 81827:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2536.357638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.706166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2540.885773] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2548.033630] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2552.907983] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2559.103166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2559.418628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.674771] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2569.650865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2573.137828] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2585.062730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.090638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2585.101347] Lustre: Skipped 1 previous similar message [ 2588.332090] Lustre: server umount lustre-OST0000 complete [ 2593.249525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2593.258374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.275309] Lustre: Skipped 1 previous similar message [ 2593.282505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2595.301554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.302304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2595.329198] Lustre: Skipped 1 previous similar message [ 2599.163417] Lustre: server umount lustre-MDT0000 complete [ 2600.423243] LustreError: 83133:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.469542] LustreError: 83133:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2603.496526] LustreError: 81807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842485 with bad export cookie 17467105366833850981 [ 2603.499026] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.507913] LustreError: 81807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2604.087037] Lustre: server umount lustre-MDT0001 complete [ 2619.642893] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2623.711920] Key type lgssc unregistered [ 2624.067412] LNet: 85363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2624.084649] LNetError: 85363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2624.096783] LNet: Removed LNI 192.168.203.128@tcp [ 2625.104774] Key type .llcrypt unregistered [ 2625.111181] Key type ._llcrypt unregistered [ 2648.946938] Key type ._llcrypt registered [ 2648.952019] Key type .llcrypt registered [ 2649.174172] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2650.505654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2650.556782] alg: No test for adler32 (adler32-zlib) [ 2651.755580] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2651.981493] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2653.631230] Key type lgssc registered [ 2654.753991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2663.155738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2663.191238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.905822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.743719] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2674.148283] LustreError: 86776:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.191335] LustreError: 86776:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2679.272808] LustreError: 86777:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.488893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.040383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.590870] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2693.531548] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2698.645173] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2704.839825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2705.286648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.356551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2710.971368] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2717.675113] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2724.835331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2724.840197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2724.858046] Lustre: Skipped 1 previous similar message [ 2724.858625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2724.874347] Lustre: Skipped 1 previous similar message [ 2726.564951] Lustre: server umount lustre-OST0000 complete [ 2733.027840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2733.036909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.050509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.073467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.077621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.095720] Lustre: Skipped 1 previous similar message [ 2736.406758] Lustre: server umount lustre-MDT0000 complete [ 2739.962628] LustreError: 86756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842621 with bad export cookie 17741544686733162182 [ 2739.966474] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2739.975191] LustreError: 86756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2740.271790] Lustre: server umount lustre-MDT0001 complete [ 2751.188791] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2753.762293] Key type lgssc unregistered [ 2754.085745] LNet: 90310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2754.103489] LNetError: 90310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2754.135351] LNet: Removed LNI 192.168.203.128@tcp [ 2754.962241] Key type .llcrypt unregistered [ 2754.965414] Key type ._llcrypt unregistered [ 2778.578879] Key type ._llcrypt registered [ 2778.585667] Key type .llcrypt registered [ 2778.754433] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2780.089607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2780.199668] alg: No test for adler32 (adler32-zlib) [ 2781.258571] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2781.610800] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2783.353174] Key type lgssc registered [ 2784.451176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2793.192428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2793.210542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.992888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.113485] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2804.196107] LustreError: 91728:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.224470] LustreError: 91728:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2809.315069] LustreError: 91729:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.425947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.820413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2814.012550] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2821.505787] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2826.822089] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2833.859171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2834.275434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.289812] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 2840.449847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2847.445310] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2858.983718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2858.995637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.012834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2861.031288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.035769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2861.036096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2861.066520] Lustre: Skipped 1 previous similar message [ 2866.145489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2866.158259] Lustre: Skipped 1 previous similar message [ 2871.271756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2871.278663] Lustre: Skipped 1 previous similar message [ 2872.287389] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2872.510351] Lustre: server umount lustre-OST0000 complete [ 2876.388533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2876.391570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.399369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2876.412654] Lustre: Skipped 1 previous similar message [ 2876.426901] Lustre: Skipped 1 previous similar message [ 2886.625449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2886.634126] Lustre: Skipped 3 previous similar messages [ 2890.207131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2890.392066] Lustre: server umount lustre-MDT0000 complete [ 2891.745858] LustreError: 94221:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.763482] LustreError: 94221:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2894.019944] LustreError: 91709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842775 with bad export cookie 1615951324295853880 [ 2894.020985] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2894.038126] LustreError: 91709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2894.398627] Lustre: server umount lustre-MDT0001 complete [ 2904.133455] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 2906.786143] Key type lgssc unregistered [ 2907.120774] LNet: 95269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2907.130615] LNetError: 95269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2907.152165] LNet: Removed LNI 192.168.203.128@tcp [ 2908.007515] Key type .llcrypt unregistered [ 2908.009556] Key type ._llcrypt unregistered [ 2930.204531] Key type ._llcrypt registered [ 2930.211491] Key type .llcrypt registered [ 2930.310877] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 2931.201348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2931.239238] alg: No test for adler32 (adler32-zlib) [ 2932.361722] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 2932.623846] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 2934.271173] Key type lgssc registered [ 2935.230681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2941.608739] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2941.627491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2957.153574] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2962.273198] LustreError: 96653:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 2989.024733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.498736] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3000.415602] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3004.851621] Lustre: server umount lustre-OST0000 complete [ 3012.159571] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 11:01:32 (1787842892) [ 3018.340383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.774335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.140643] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3023.845982] LustreError: 98305:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.867231] LustreError: 98305:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3028.962635] LustreError: 98304:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3031.695542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3032.097866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3036.145250] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3043.730819] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3049.467827] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3055.936253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3056.284264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.027873] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3063.488326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3069.997634] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3083.232306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3083.238849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3083.251190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3084.259075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3084.270623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3084.286726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.961163] Lustre: server umount lustre-OST0000 complete [ 3093.483759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3093.488419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.495260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3093.505521] Lustre: Skipped 1 previous similar message [ 3098.737518] Lustre: server umount lustre-MDT0000 complete [ 3099.624922] LustreError: 100143:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.658675] LustreError: 100143:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3102.913666] LustreError: 100954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842984 with bad export cookie 7091966237832977773 [ 3102.921305] LustreError: 100954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3102.921710] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3103.335920] Lustre: server umount lustre-MDT0001 complete [ 3113.549204] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3116.178179] Key type lgssc unregistered [ 3116.494820] LNet: 102001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3116.501867] LNetError: 102001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3116.517481] LNet: Removed LNI 192.168.203.128@tcp [ 3117.332107] Key type .llcrypt unregistered [ 3117.339262] Key type ._llcrypt unregistered [ 3150.776408] Key type ._llcrypt registered [ 3150.779089] Key type .llcrypt registered [ 3150.878996] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3151.746256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3151.783616] alg: No test for adler32 (adler32-zlib) [ 3152.790890] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 3153.083388] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3154.775241] Key type lgssc registered [ 3155.886789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3162.658597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3162.685076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.032629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.079927] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3173.352838] LustreError: 104163:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.367056] LustreError: 104163:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3174.882814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.241702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3178.826771] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3184.773587] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3189.598120] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3195.542576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3195.871215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3200.853882] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3207.518435] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3208.186587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3216.352209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3216.364946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.386472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3218.400583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3218.401355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3218.408322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3221.664936] Lustre: server umount lustre-OST0000 complete [ 3226.592252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3226.604519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.617913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3226.627170] Lustre: Skipped 1 previous similar message [ 3228.643606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3228.644047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.780728] Lustre: server umount lustre-MDT0000 complete [ 3233.762684] LustreError: 104163:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.781511] LustreError: 104163:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3235.330568] LustreError: 104145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843117 with bad export cookie 8438640003368421806 [ 3235.331929] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3235.351204] LustreError: 104145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3235.747407] Lustre: server umount lustre-MDT0001 complete [ 3246.971670] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3249.761871] Key type lgssc unregistered [ 3250.117934] LNet: 107698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3250.128421] LNetError: 107698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3250.142945] LNet: Removed LNI 192.168.203.128@tcp [ 3250.919839] Key type .llcrypt unregistered [ 3250.924209] Key type ._llcrypt unregistered [ 3267.410908] Key type ._llcrypt registered [ 3267.412929] Key type .llcrypt registered [ 3267.495214] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3268.285379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3268.367519] alg: No test for adler32 (adler32-zlib) [ 3269.328226] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 3269.554674] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3271.231660] Key type lgssc registered [ 3272.398743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3280.422979] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3280.438889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3295.969957] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3301.088139] LustreError: 108944:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 3327.775330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3334.519806] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3341.578931] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3348.120386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.501247] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3348.523544] LustreError: 109967:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3348.530724] LustreError: 109967:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3348.543697] LustreError: 109967:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3348.650200] Lustre: server umount lustre-MDT0000 complete [ 3348.654623] LustreError: 109967:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3369.953827] Lustre: 108282:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843235/real 1787843235] req@00000000132558c3 x1874689419906304/t0(0) o400->MGC192.168.203.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787843251 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3369.990935] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3370.011559] LustreError: 108956:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3390.562333] Lustre: server umount lustre-OST0000 complete [ 3399.318495] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 3408.940986] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3445.332616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3445.681397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3445.705823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3445.757745] Lustre: lustre-MDT0000: new disk, initializing [ 3445.876283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.898511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3450.444818] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3460.896477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.035746] Lustre: 112762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3461.123043] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3461.126751] Lustre: Skipped 1 previous similar message [ 3461.222539] Lustre: lustre-MDT0001: new disk, initializing [ 3461.313996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3461.421465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3461.443968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3467.784572] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3477.034439] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3483.314581] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3490.441174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3490.759029] Lustre: lustre-OST0000: new disk, initializing [ 3490.766515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3490.785798] Lustre: 114728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3490.887406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.896418] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3498.565910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3498.580588] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3498.674177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3507.101603] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3511.801852] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3512.057437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3516.508044] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3516.712235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3519.459677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.461158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3519.468315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3519.468381] Lustre: Skipped 1 previous similar message [ 3519.498262] LustreError: Skipped 1 previous similar message [ 3524.578096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3524.595243] Lustre: Skipped 2 previous similar messages [ 3524.789797] Lustre: server umount lustre-OST0000 complete [ 3529.705437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.717882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.727460] Lustre: Skipped 1 previous similar message [ 3529.751618] Lustre: Skipped 1 previous similar message [ 3534.789385] Lustre: server umount lustre-MDT0000 complete [ 3538.636586] LustreError: 112754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843420 with bad export cookie 12091855196088841508 [ 3538.639233] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3538.654061] LustreError: 112754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3538.986271] Lustre: server umount lustre-MDT0001 complete [ 3547.554559] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 11:10:28 (1787843428) [ 3551.204987] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3569.651911] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 3583.060449] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3621.037480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.347217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3621.387336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3621.477410] Lustre: lustre-MDT0000: new disk, initializing [ 3621.582857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3621.607082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3626.574455] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3637.424062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3637.566481] Lustre: 119632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3637.625021] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3637.629190] Lustre: Skipped 1 previous similar message [ 3637.714584] Lustre: lustre-MDT0001: new disk, initializing [ 3637.806269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3637.851878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3637.861196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3642.056296] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3651.290289] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3656.868236] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3663.850964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3664.029923] Lustre: lustre-OST0000: new disk, initializing [ 3664.033273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3664.042737] Lustre: 121601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3664.117842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3670.586887] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3671.144074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3671.159383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3671.266260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3680.485449] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3685.860731] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3686.205606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3690.938599] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3691.112726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3697.126414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3697.128693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3697.139179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3697.161926] Lustre: Skipped 1 previous similar message [ 3702.246198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3702.258200] Lustre: Skipped 2 previous similar messages [ 3707.872310] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3708.140170] Lustre: server umount lustre-OST0000 complete [ 3712.480549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3712.494581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3712.501498] Lustre: Skipped 3 previous similar messages [ 3718.418905] Lustre: server umount lustre-MDT0000 complete [ 3722.182067] LustreError: 121600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843604 with bad export cookie 12091855196088842397 [ 3722.191319] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3722.194436] LustreError: 121600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3722.592771] Lustre: server umount lustre-MDT0001 complete [ 3731.753240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.089445] LustreError: 123818:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3732.110158] LustreError: 123818:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3732.179648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3736.205812] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3737.569929] LustreError: 123819:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3742.700831] LustreError: 123818:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3744.733302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.839804] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3756.902705] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3763.434527] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3770.268075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3777.201545] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3785.897502] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3795.813872] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3803.629164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.641138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.671934] Lustre: Skipped 1 previous similar message [ 3803.686044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.692354] Lustre: Skipped 2 previous similar messages [ 3806.798628] Lustre: server umount lustre-OST0000 complete [ 3811.815042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3811.820285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.839347] Lustre: Skipped 1 previous similar message [ 3817.018097] Lustre: server umount lustre-MDT0000 complete [ 3820.002206] LustreError: 123813:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.028667] LustreError: 123813:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3821.420631] LustreError: 123800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843703 with bad export cookie 12091855196088843279 [ 3821.426946] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3821.438225] LustreError: 123800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3821.808325] Lustre: server umount lustre-MDT0001 complete [ 3836.372458] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 3839.429784] Key type lgssc unregistered [ 3839.660505] LNet: 127454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3839.668928] LNetError: 127454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3839.682531] LNet: Removed LNI 192.168.203.128@tcp [ 3840.490122] Key type .llcrypt unregistered [ 3840.492083] Key type ._llcrypt unregistered [ 3857.707150] Key type ._llcrypt registered [ 3857.709851] Key type .llcrypt registered [ 3857.821338] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 3872.218982] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 3873.497138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3873.571140] alg: No test for adler32 (adler32-zlib) [ 3874.686985] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 3874.956360] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 3876.671204] Key type lgssc registered [ 3877.817691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3912.016118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3912.101605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3913.350759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3913.413792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3913.482723] Lustre: lustre-MDT0000: new disk, initializing [ 3913.593154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.605220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3918.272166] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3928.905835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.999484] Lustre: 130606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3929.050470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3929.054070] Lustre: Skipped 1 previous similar message [ 3929.115555] Lustre: lustre-MDT0001: new disk, initializing [ 3929.187477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3929.224492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3929.240187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3934.238962] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3942.652657] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3947.957893] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3954.177981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3954.495356] Lustre: lustre-OST0000: new disk, initializing [ 3954.499332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3954.504540] Lustre: 132573:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3954.595077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.533248] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 3962.936495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3962.948472] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3963.089335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3970.405964] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3976.031806] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3976.351679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3982.294331] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3982.494613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3988.960971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3988.964683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.975566] LustreError: Skipped 1 previous similar message [ 3989.008981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3994.083351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3994.089163] Lustre: Skipped 2 previous similar messages [ 3999.201570] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3999.222535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3999.253208] Lustre: Skipped 1 previous similar message [ 3999.520947] Lustre: server umount lustre-OST0000 complete [ 4004.322513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.324847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4004.329768] Lustre: Skipped 2 previous similar messages [ 4004.333570] Lustre: Skipped 1 previous similar message [ 4009.440431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4009.443880] Lustre: Skipped 1 previous similar message [ 4010.019364] Lustre: server umount lustre-MDT0000 complete [ 4014.320228] LustreError: 131401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843896 with bad export cookie 8420261821366156204 [ 4014.324363] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4014.330845] LustreError: 131401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4014.855682] Lustre: server umount lustre-MDT0001 complete [ 4024.026086] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 11:18:24 (1787843904) [ 4030.305845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.719754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4034.827805] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4036.065256] LustreError: 134985:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4036.084407] LustreError: 134985:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4040.168587] LustreError: 134984:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4042.968931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.367294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4048.058084] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4055.568761] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4060.287408] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4063.721440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4063.739635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4063.752588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4066.282781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.264369] Lustre: server umount lustre-MDT0000 complete [ 4068.322851] LustreError: 134984:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.341494] LustreError: 134984:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4071.677387] LustreError: 134966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843953 with bad export cookie 8420261821366157086 [ 4071.692695] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4072.182478] Lustre: server umount lustre-MDT0001 complete [ 4080.854616] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 11:19:21 (1787843961) [ 4086.968181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4102.498474] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4107.616081] LustreError: 137399:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 4133.343761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.838298] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4145.839851] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4147.699845] Lustre: server umount lustre-OST0000 complete [ 4154.719977] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 11:20:35 (1787844035) [ 4160.382566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.706980] LustreError: 138843:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4160.771889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4164.386245] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4166.116056] LustreError: 138844:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.360240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4175.355176] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4181.578390] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4187.374730] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4193.350763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.681349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4193.691703] Lustre: Skipped 1 previous similar message [ 4198.947117] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4205.728454] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4212.704962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4212.709576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.729348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4212.738312] Lustre: Skipped 2 previous similar messages [ 4218.087445] Lustre: server umount lustre-MDT0000 complete [ 4221.171147] LustreError: 138825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844103 with bad export cookie 8420261821366157744 [ 4221.193718] LustreError: 138825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4221.208811] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4221.408861] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.413939] LustreError: 141187:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.433454] Lustre: Skipped 2 previous similar messages [ 4221.456344] LustreError: 141187:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4221.534806] Lustre: server umount lustre-MDT0001 complete [ 4231.254616] Lustre: server umount lustre-OST0000 complete [ 4239.266597] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 11:21:59 (1787844119) [ 4245.095760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4245.446452] LustreError: 142498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4249.720469] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4257.840072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4258.205754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4258.216834] Lustre: Skipped 1 previous similar message [ 4263.143821] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4272.036584] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4278.899070] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4286.581866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4290.239254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4294.602509] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4302.793811] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4308.444566] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4308.758345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4314.832595] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4315.212337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4319.201839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4319.224429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4319.229034] Lustre: Skipped 1 previous similar message [ 4319.240046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4319.254362] Lustre: Skipped 4 previous similar messages [ 4323.655352] Lustre: server umount lustre-OST0000 complete [ 4333.948331] Lustre: server umount lustre-MDT0000 complete [ 4334.565773] LustreError: 142498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.583405] LustreError: 142498:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4338.125548] LustreError: 142478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844220 with bad export cookie 8420261821366158822 [ 4338.142639] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4338.145455] LustreError: 142478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4338.496664] Lustre: server umount lustre-MDT0001 complete [ 4346.909144] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 11:23:47 (1787844227) [ 4353.494932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4369.248780] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4374.367931] LustreError: 146657:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 4400.095293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4400.107484] Lustre: Skipped 1 previous similar message [ 4405.943856] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4412.845779] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4420.191503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.726512] LustreError: 146682:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4420.935304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4425.887707] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4433.908269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4438.204270] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4446.409522] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4452.231852] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4456.638420] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4456.877735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4460.719215] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4460.897641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4465.120826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4465.126567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.138306] Lustre: Skipped 2 previous similar messages [ 4465.144278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4465.147486] Lustre: Skipped 2 previous similar messages [ 4469.001685] Lustre: server umount lustre-MDT0000 complete [ 4472.211224] LustreError: 146663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844354 with bad export cookie 8420261821366160068 [ 4472.216363] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4472.232766] LustreError: 146663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4472.636750] Lustre: server umount lustre-MDT0001 complete [ 4486.115874] Lustre: server umount lustre-OST0000 complete [ 4493.427918] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 11:26:14 (1787844374) [ 4499.838845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4515.487957] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4520.609716] LustreError: 150852:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 4552.572785] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4560.664369] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4567.511763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.149718] LustreError: 150877:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4568.174757] LustreError: 150877:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 4568.298162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4573.037176] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4580.905337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.519981] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4593.582406] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4599.603797] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4606.365287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4606.569569] Lustre: lustre-OST0001: new disk, initializing [ 4606.575682] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4606.589623] Lustre: 153728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4614.159747] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4614.738014] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4614.753089] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4614.885652] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4622.601902] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4627.136249] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4627.325614] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4632.823717] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4633.036315] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4635.628048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.644130] Lustre: Skipped 2 previous similar messages [ 4635.667835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4635.678204] Lustre: Skipped 3 previous similar messages [ 4640.224642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4641.809879] Lustre: server umount lustre-OST0000 complete [ 4653.032494] Lustre: server umount lustre-OST0001 complete [ 4658.144292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4658.157384] LustreError: Skipped 2 previous similar messages [ 4663.777079] Lustre: server umount lustre-MDT0000 complete [ 4668.272506] LustreError: 150858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844550 with bad export cookie 8420261821366161020 [ 4668.275574] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4668.283435] LustreError: 150858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4668.965664] Lustre: server umount lustre-MDT0001 complete [ 4691.846561] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 11:29:32 (1787844572) [ 4693.272578] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4695.021459] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4696.825062] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 11:29:37 (1787844577) [ 4704.618050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.004604] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4705.061845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4705.366147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4705.377613] Lustre: Skipped 6 previous similar messages [ 4709.918409] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4721.001620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.074659] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4721.139455] Lustre: 157285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4726.893397] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4735.477758] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4741.364445] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4747.353748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4747.573953] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4753.640060] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4756.114792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:193) [ 4761.744650] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4766.102405] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4766.332776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4770.138687] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4770.334433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4776.417407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4776.426836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.438602] Lustre: Skipped 5 previous similar messages [ 4778.411628] Lustre: server umount lustre-OST0000 complete [ 4800.039199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4806.449536] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4813.985302] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4816.825371] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4816.842829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4816.843742] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4816.872256] Lustre: Skipped 1 previous similar message [ 4818.823536] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4818.989620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4823.399955] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4823.693501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4829.434928] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93ed5900f800.ost_server_uuid 50 [ 4830.935646] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93ed5900f800.ost_server_uuid in FULL state after 0 sec [ 4831.103062] LustreError: 157294:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.28@tcp) returned error from blocking AST (req@00000000b87e8075 x1874690054550400 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000bdc2a87a/0x74dac7839a5057d6 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.28@tcp remote: 0xe4f2f703cfe83cfb expref: 5 pid: 159444 timeout: 4931 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4831.143534] LustreError: lustre-MDT0000: A client on nid 192.168.203.28@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4831.157434] LustreError: 157282:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.28@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000bdc2a87a/0x74dac7839a5057d6 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.28@tcp remote: 0xe4f2f703cfe83cfb expref: 6 pid: 159444 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4841.595511] Lustre: server umount lustre-OST0000 complete [ 4852.016099] Lustre: server umount lustre-MDT0000 complete [ 4852.713235] LustreError: 157297:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.746072] LustreError: 157297:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 4856.144636] LustreError: 157279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844738 with bad export cookie 8420261821366162196 [ 4856.149973] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4856.163465] LustreError: 157279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4862.578511] Lustre: server umount lustre-MDT0001 complete [ 4873.719666] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 4876.170818] Key type lgssc unregistered [ 4876.400833] LNet: 163401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4876.406838] LNetError: 163401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4876.422656] LNet: Removed LNI 192.168.203.128@tcp [ 4877.207519] Key type .llcrypt unregistered [ 4877.209406] Key type ._llcrypt unregistered [ 4900.469252] Key type ._llcrypt registered [ 4900.473577] Key type .llcrypt registered [ 4900.598540] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 4901.801819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4901.828732] alg: No test for adler32 (adler32-zlib) [ 4902.924555] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 4903.190448] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 4904.855576] Key type lgssc registered [ 4906.203354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4913.538154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4913.584963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4914.992857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.881592] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4924.386912] LustreError: 164813:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4924.406720] LustreError: 164813:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4927.635663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4928.119963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4932.843616] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4941.847632] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4947.962846] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4954.959848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4955.261215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4959.347219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4960.799221] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 4967.833309] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4977.572646] Lustre: Failing over lustre-MDT0000 [ 4977.871856] Lustre: server umount lustre-MDT0000 complete [ 4979.173382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4979.180289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4979.197088] LustreError: 164813:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4979.246285] Lustre: Skipped 2 previous similar messages [ 4984.289655] LustreError: 167304:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4984.351055] LustreError: 167304:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4989.410813] LustreError: 164810:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4989.431660] LustreError: 164810:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4990.238834] LustreError: 164793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.28@tcp arrived at 1787844872 with bad export cookie 15242095807963070301 [ 4994.552091] LustreError: 164814:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4994.569863] LustreError: 164814:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4995.615406] Lustre: 164125:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844861/real 1787844861] req@00000000487bbc8f x1874691133302656/t0(0) o400->MGC192.168.203.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787844877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4995.639657] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4997.302907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.803230] LustreError: 164813:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5005.823710] LustreError: 164813:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 5006.033350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5006.078394] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5010.384872] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5011.425758] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5011.436194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5012.301896] Lustre: lustre-MDT0000: Denying connection for new client f209ecf1-b0ad-46f5-96c1-b23af5e97e26 (at 192.168.203.28@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5021.677294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.682727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5021.695566] Lustre: Skipped 1 previous similar message [ 5021.718737] Lustre: Skipped 1 previous similar message [ 5026.595183] Lustre: server umount lustre-OST0000 complete [ 5030.461584] LustreError: 168542:0:(ldlm_lib.c:3067:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5030.483532] Lustre: 167890:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5030.498916] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5030.520749] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5030.543650] LustreError: 167890:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b601ce84 x1874691133323264/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5030.582918] LustreError: 167890:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5030.600361] LustreError: 167890:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5030.624328] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5031.019824] Lustre: server umount lustre-MDT0000 complete [ 5031.915299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5031.939612] LustreError: 164809:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5034.603615] LustreError: 164795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844916 with bad export cookie 15242095807963070406 [ 5034.603915] LustreError: 164124:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5034.620473] LustreError: 164795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5034.640780] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5037.014234] Lustre: server umount lustre-MDT0001 complete [ 5048.507234] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 5051.230128] Key type lgssc unregistered [ 5051.490137] LNet: 169297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5051.499408] LNetError: 169297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5051.521596] LNet: Removed LNI 192.168.203.128@tcp [ 5052.415179] Key type .llcrypt unregistered [ 5052.417090] Key type ._llcrypt unregistered [ 5074.583142] Key type ._llcrypt registered [ 5074.585226] Key type .llcrypt registered [ 5074.694799] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5075.997362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5076.031697] alg: No test for adler32 (adler32-zlib) [ 5077.253949] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 5077.622975] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 5079.303465] Key type lgssc registered [ 5080.422268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5088.173641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5088.187170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.652276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5093.896771] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5100.007439] LustreError: 170703:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5100.023735] LustreError: 170703:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5102.848539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5103.228575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5108.060135] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5115.298697] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5121.546171] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5128.617314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5128.938797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5135.116796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 5136.147707] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5144.160501] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5153.783555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5153.786633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5153.802096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5153.820451] Lustre: Skipped 1 previous similar message [ 5153.830634] Lustre: Skipped 1 previous similar message [ 5157.256126] Lustre: server umount lustre-OST0000 complete [ 5164.002157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.009887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5164.019267] Lustre: Skipped 1 previous similar message [ 5164.033903] Lustre: Skipped 1 previous similar message [ 5167.336784] Lustre: server umount lustre-MDT0000 complete [ 5169.121237] LustreError: 171432:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5169.135800] LustreError: 171432:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5170.476808] LustreError: 172548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845052 with bad export cookie 4794587344610683007 [ 5170.482429] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5170.485504] LustreError: 172548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5170.825945] Lustre: server umount lustre-MDT0001 complete [ 5179.627814] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 5182.180833] Key type lgssc unregistered [ 5182.504727] LNet: 174239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5182.512508] LNetError: 174239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5182.543834] LNet: Removed LNI 192.168.203.128@tcp [ 5183.294980] Key type .llcrypt unregistered [ 5183.300219] Key type ._llcrypt unregistered [ 5215.744314] Key type ._llcrypt registered [ 5215.746881] Key type .llcrypt registered [ 5215.834752] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5217.033179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5217.048227] alg: No test for adler32 (adler32-zlib) [ 5218.127751] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 5218.362887] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 5220.031159] Key type lgssc registered [ 5221.220840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5228.083242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5228.097350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.544268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5234.142376] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5239.782153] LustreError: 175941:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5239.812162] LustreError: 175941:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5242.582592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5242.923688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5247.087899] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5255.026063] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5260.832939] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5268.260689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5268.625379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5274.753215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 5274.823608] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5282.834212] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5295.585645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5295.591597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.602491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5298.656756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5298.661871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5298.662468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.514728] Lustre: server umount lustre-OST0000 complete [ 5303.776454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.779193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5303.790711] Lustre: Skipped 1 previous similar message [ 5303.793806] Lustre: Skipped 1 previous similar message [ 5308.900040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5308.907223] Lustre: Skipped 2 previous similar messages [ 5309.828563] Lustre: server umount lustre-MDT0000 complete [ 5313.406720] LustreError: 175922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845195 with bad export cookie 3008075411185666898 [ 5313.421063] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5313.424996] LustreError: 175922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5313.850194] Lustre: server umount lustre-MDT0001 complete [ 5328.216734] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 5331.905047] Key type lgssc unregistered [ 5332.440108] LNet: 179477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5332.464377] LNetError: 179477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5332.505051] LNet: Removed LNI 192.168.203.128@tcp [ 5334.325234] Key type .llcrypt unregistered [ 5334.330337] Key type ._llcrypt unregistered [ 5362.405509] Key type ._llcrypt registered [ 5362.408117] Key type .llcrypt registered [ 5362.555840] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5363.663139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5363.744304] alg: No test for adler32 (adler32-zlib) [ 5364.805705] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 5365.135719] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 5366.826408] Key type lgssc registered [ 5367.897667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5378.586425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5378.605472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.988093] Lustre: *** cfs_fail_loc=135, val=0*** [ 5380.064618] LustreError: 180999:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5380.074771] LustreError: 180999:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.128@tcp: cfg command failed: rc = -2 [ 5380.081313] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5380.081313] [ 5380.090255] LustreError: MGC192.168.203.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5380.098935] LustreError: 180960:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5380.108869] LustreError: 180960:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5380.114870] LustreError: 180960:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5380.119272] LustreError: 180960:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5380.165868] Lustre: server umount lustre-MDT0000 complete [ 5380.169462] LustreError: 180960:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5385.672378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5386.080332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5389.674520] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5395.427482] LustreError: 181370:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5395.447574] LustreError: 181370:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5397.701391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5398.147463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5403.167282] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5410.031266] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5415.016180] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5418.467622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5418.473743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5418.505499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.028815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.029371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5423.140302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5423.143301] Lustre: Skipped 1 previous similar message [ 5423.726480] Lustre: server umount lustre-MDT0000 complete [ 5427.120048] LustreError: 181352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845309 with bad export cookie 16350661919855837761 [ 5427.123788] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5427.139319] LustreError: 181352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5427.530643] Lustre: server umount lustre-MDT0001 complete [ 5438.122972] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 5441.527664] Key type lgssc unregistered [ 5441.893303] LNet: 183694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5441.905034] LNetError: 183694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5441.918636] LNet: Removed LNI 192.168.203.128@tcp [ 5442.898532] Key type .llcrypt unregistered [ 5442.901732] Key type ._llcrypt unregistered [ 5468.350874] Key type ._llcrypt registered [ 5468.354191] Key type .llcrypt registered [ 5468.499647] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 5468.775312] Key type .llcrypt unregistered [ 5468.778297] Key type ._llcrypt unregistered [ 5486.270956] Key type ._llcrypt registered [ 5486.278256] Key type .llcrypt registered [ 5486.385566] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5487.582879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5487.625265] alg: No test for adler32 (adler32-zlib) [ 5488.670588] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 5488.884609] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 5490.569425] Key type lgssc registered [ 5491.699629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5498.476322] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5498.489964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5514.015675] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5519.135374] LustreError: 185832:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.128@tcp: failed processing log, type 4: rc = -110 [ 5545.823175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5552.636820] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5560.762979] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5567.550206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5567.659762] LustreError: 185168:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f90f60a1 x1874691746695424/t0(0) o250->MGC192.168.203.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5567.941518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.977092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:321) [ 5572.080544] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5573.090836] LustreError: 185857:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.103470] LustreError: 185857:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5578.212936] LustreError: 185858:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5580.310737] LustreError: 186882:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5580.340419] LustreError: 186882:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5582.457256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5583.107366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5589.000914] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5595.730538] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5601.102864] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5604.205530] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5614.055087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5614.056021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5614.069459] LustreError: Skipped 1 previous similar message [ 5614.072404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5614.109609] Lustre: Skipped 1 previous similar message [ 5619.178453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5619.186849] Lustre: Skipped 1 previous similar message [ 5624.293762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5624.304644] Lustre: Skipped 1 previous similar message [ 5628.389728] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5628.591132] Lustre: server umount lustre-OST0000 complete [ 5634.529239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5634.533140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5634.570631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5634.586216] Lustre: Skipped 2 previous similar messages [ 5639.668265] LustreError: 186882:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.706499] LustreError: 186882:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5639.799514] Lustre: server umount lustre-MDT0000 complete [ 5643.579584] LustreError: 185840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845525 with bad export cookie 5822947028934861396 [ 5643.583375] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5643.616093] LustreError: 185840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5644.117794] Lustre: server umount lustre-MDT0001 complete [ 5653.808659] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 5656.295880] Key type lgssc unregistered [ 5656.593220] LNet: 189811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5656.601799] LNetError: 189811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5656.622871] LNet: Removed LNI 192.168.203.128@tcp [ 5657.359623] Key type .llcrypt unregistered [ 5657.372213] Key type ._llcrypt unregistered [ 5680.988445] Key type ._llcrypt registered [ 5681.000041] Key type .llcrypt registered [ 5681.128772] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5682.609451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5682.667616] alg: No test for adler32 (adler32-zlib) [ 5683.736030] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 5684.048980] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 5685.751239] Key type lgssc registered [ 5686.968627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5694.921579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5694.965378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5696.398345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5701.075936] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5706.729427] LustreError: 191223:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5706.752362] LustreError: 191223:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5710.081230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5710.522788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5715.054129] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5722.222765] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5727.783059] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5735.618856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5736.049238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5742.217310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:353) [ 5744.383393] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5753.911361] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5759.339434] Lustre: Failing over lustre-MDT0000 [ 5759.752506] Lustre: server umount lustre-MDT0000 complete [ 5761.192916] LustreError: 192662:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5761.508143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5761.520251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.065882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.070430] LustreError: 191950:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.090551] Lustre: Skipped 1 previous similar message [ 5763.104162] LustreError: 191950:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5766.215121] LustreError: 192662:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.342758] LustreError: 191218:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.368254] LustreError: 191218:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5778.338400] Lustre: 190533:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787845644/real 1787845644] req@00000000b0f3df7f x1874691951193344/t0(0) o400->MGC192.168.203.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787845660 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5778.359713] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5779.363605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.577136] LustreError: 193523:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5781.608439] LustreError: 193523:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 5788.644820] LustreError: 190532:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007ee4d876 x1874691951198208/t0(0) o250->MGC192.168.203.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5789.018729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5789.075222] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5792.843018] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5793.844421] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5794.281556] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5794.305181] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5794.353846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:385) [ 5798.118981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5805.829943] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5820.385124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5820.388145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5820.398586] LustreError: Skipped 1 previous similar message [ 5820.427868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.449949] Lustre: server umount lustre-OST0000 complete [ 5830.112313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5830.115707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5830.123628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5830.147867] Lustre: Skipped 1 previous similar message [ 5830.163186] Lustre: Skipped 1 previous similar message [ 5832.862380] Lustre: server umount lustre-MDT0000 complete [ 5835.233657] LustreError: 191950:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5835.252860] LustreError: 191950:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 5836.867058] LustreError: 191203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845718 with bad export cookie 15534705440402805270 [ 5836.873425] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5836.880949] LustreError: 191203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5846.722625] Lustre: server umount lustre-MDT0001 complete [ 5858.621495] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 5861.718583] Key type lgssc unregistered [ 5862.118454] LNet: 196138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5862.132440] LNetError: 196138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5862.166253] LNet: Removed LNI 192.168.203.128@tcp [ 5863.428710] Key type .llcrypt unregistered [ 5863.431225] Key type ._llcrypt unregistered [ 5891.046276] Key type ._llcrypt registered [ 5891.048673] Key type .llcrypt registered [ 5891.172469] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 5892.253680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5892.295100] alg: No test for adler32 (adler32-zlib) [ 5893.468109] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 5893.709404] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 5895.343166] Key type lgssc registered [ 5896.429495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5905.087863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5905.133107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.646387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.332214] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5912.032599] LustreError: 197551:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5912.053069] LustreError: 197551:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5916.134219] LustreError: 197552:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.709355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.034869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5924.541077] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5931.683725] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5937.298441] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5943.258121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5943.798418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5949.542833] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 5952.008553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:417) [ 5956.497643] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5959.172076] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5964.731344] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5983.471396] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5990.880892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5990.897228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.913637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5992.928200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5992.947211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.975983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5996.633963] Lustre: server umount lustre-OST0000 complete [ 6002.152142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6002.155416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6002.171089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6002.173428] Lustre: Skipped 1 previous similar message [ 6008.293097] LustreError: 198279:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6008.354761] Lustre: server umount lustre-MDT0000 complete [ 6012.040016] LustreError: 197531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845893 with bad export cookie 4738701366942124967 [ 6012.047234] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6012.052780] LustreError: 197531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6012.368054] Lustre: server umount lustre-MDT0001 complete [ 6022.967184] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6025.456051] Key type lgssc unregistered [ 6025.749627] LNet: 201428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6025.760075] LNetError: 201428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6025.785903] LNet: Removed LNI 192.168.203.128@tcp [ 6026.605222] Key type .llcrypt unregistered [ 6026.610356] Key type ._llcrypt unregistered [ 6042.167437] Key type ._llcrypt registered [ 6042.169829] Key type .llcrypt registered [ 6042.243942] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6043.231849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6043.270365] alg: No test for adler32 (adler32-zlib) [ 6044.307976] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 6044.513332] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6046.199224] Key type lgssc registered [ 6047.338939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6054.246051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6054.268576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6055.739175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.206286] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6065.124090] LustreError: 202689:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6065.142959] LustreError: 202689:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6069.268766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.693224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6073.832332] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6080.710597] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6087.305951] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6093.297932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6093.623285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.031872] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6100.792858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:449) [ 6107.118522] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6113.605969] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6117.281340] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6130.462398] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6135.780273] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6146.020889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6146.028552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.057975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6147.041667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6147.051570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6149.800360] Lustre: server umount lustre-OST0000 complete [ 6156.768370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6156.784787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.797462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6156.802343] Lustre: Skipped 1 previous similar message [ 6159.289372] Lustre: server umount lustre-MDT0000 complete [ 6162.401033] LustreError: 202689:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.437416] LustreError: 202689:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6162.822963] LustreError: 205367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846044 with bad export cookie 6025063234740875241 [ 6162.831991] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6162.842610] LustreError: 205367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6163.213899] Lustre: server umount lustre-MDT0001 complete [ 6173.908982] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6176.655745] Key type lgssc unregistered [ 6176.945643] LNet: 207473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6176.952989] LNetError: 207473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6178.025180] LNet: Removed LNI 192.168.203.128@tcp [ 6178.832339] Key type .llcrypt unregistered [ 6178.846386] Key type ._llcrypt unregistered [ 6203.363289] Key type ._llcrypt registered [ 6203.366258] Key type .llcrypt registered [ 6203.458597] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6204.118643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6204.252141] alg: No test for adler32 (adler32-zlib) [ 6205.426554] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 6205.689688] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6207.327217] Key type lgssc registered [ 6208.547263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6218.553331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6218.599046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6220.291135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6225.494565] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6230.499174] LustreError: 208886:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6230.521347] LustreError: 208886:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6234.679597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.050787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6240.149740] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6248.801428] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6254.909428] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6261.735124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6262.128160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.211893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:481) [ 6267.862772] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6275.220936] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6289.898791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6289.904453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.910372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6289.919338] Lustre: Skipped 1 previous similar message [ 6289.927809] Lustre: Skipped 1 previous similar message [ 6295.012721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6295.018988] Lustre: Skipped 1 previous similar message [ 6295.577355] Lustre: server umount lustre-OST0000 complete [ 6300.133435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.142873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6300.144834] Lustre: Skipped 1 previous similar message [ 6300.155488] Lustre: Skipped 1 previous similar message [ 6305.227601] Lustre: server umount lustre-MDT0000 complete [ 6305.249326] LustreError: 209615:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.273164] LustreError: 209615:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6308.976303] LustreError: 208867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846190 with bad export cookie 15951736683293271309 [ 6308.977640] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6308.995272] LustreError: 208867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6309.477915] Lustre: server umount lustre-MDT0001 complete [ 6319.492875] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6322.075983] Key type lgssc unregistered [ 6322.369871] LNet: 212573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6322.376525] LNetError: 212573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6322.397647] LNet: Removed LNI 192.168.203.128@tcp [ 6323.135460] Key type .llcrypt unregistered [ 6323.142084] Key type ._llcrypt unregistered [ 6342.447625] Key type ._llcrypt registered [ 6342.449193] Key type .llcrypt registered [ 6342.601091] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6343.967409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6344.008886] alg: No test for adler32 (adler32-zlib) [ 6345.149282] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 6345.423583] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6347.112044] Key type lgssc registered [ 6348.296595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6356.378931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6356.394986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6357.969262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6362.514397] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6367.202616] LustreError: 213840:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6367.225255] LustreError: 213840:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6371.041381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.431896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6375.728861] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6383.810381] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6389.155050] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6395.741061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6395.957754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6401.986696] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6402.107370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:513) [ 6410.863538] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6432.742623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6432.765686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6432.768814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6432.786499] Lustre: Skipped 1 previous similar message [ 6435.411439] Lustre: server umount lustre-OST0000 complete [ 6442.978714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6442.983236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6442.983919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6442.995157] Lustre: Skipped 2 previous similar messages [ 6443.017234] Lustre: Skipped 1 previous similar message [ 6445.357863] Lustre: server umount lustre-MDT0000 complete [ 6448.098783] LustreError: 213840:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6448.127247] LustreError: 213840:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6449.210336] LustreError: 213820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846331 with bad export cookie 3653816366589703686 [ 6449.214991] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6449.230332] LustreError: 213820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6449.545697] Lustre: server umount lustre-MDT0001 complete [ 6460.847801] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6463.771597] Key type lgssc unregistered [ 6464.049342] LNet: 217527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6464.061879] LNetError: 217527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6465.136401] LNet: Removed LNI 192.168.203.128@tcp [ 6466.094575] Key type .llcrypt unregistered [ 6466.097302] Key type ._llcrypt unregistered [ 6485.484459] Key type ._llcrypt registered [ 6485.487989] Key type .llcrypt registered [ 6485.591266] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6486.495604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6486.624091] alg: No test for adler32 (adler32-zlib) [ 6487.706635] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 6488.007242] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6489.753056] Key type lgssc registered [ 6490.792662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6498.662810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6498.693471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6500.222225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6504.429248] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6510.569816] LustreError: 218795:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6510.592289] LustreError: 218795:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6513.029498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6513.530795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6517.926663] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6525.158304] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6530.380549] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6537.213030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6537.595371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6543.551337] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6545.784190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:545) [ 6551.356800] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6559.086692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6559.143429] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6559.401174] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6565.794552] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6573.118804] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6574.556121] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6586.403647] Lustre: Permanently deactivating lustre-OST0001 [ 6586.420472] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6593.642935] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6600.506118] Lustre: Permanently reactivating lustre-OST0001 [ 6600.517801] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6600.525565] Lustre: Skipped 2 previous similar messages [ 6607.439496] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6607.460559] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6607.469899] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6607.479883] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6607.515297] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6616.555912] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6616.560803] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6616.570396] Lustre: Skipped 2 previous similar messages [ 6616.585146] Lustre: Skipped 1 previous similar message [ 6620.609994] Lustre: server umount lustre-OST0001 complete [ 6621.679576] LustreError: 220617:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6621.692128] LustreError: 220617:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6625.760894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6625.772041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6625.790307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6626.790259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6626.793969] LustreError: 220617:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6626.808205] Lustre: Skipped 1 previous similar message [ 6626.844672] LustreError: 220617:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6630.173438] Lustre: server umount lustre-OST0000 complete [ 6636.513060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6636.519547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.529361] Lustre: Skipped 1 previous similar message [ 6636.542483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6640.503481] Lustre: server umount lustre-MDT0000 complete [ 6642.164189] LustreError: 219829:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6642.185585] LustreError: 219829:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6643.960179] LustreError: 221641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846525 with bad export cookie 8386217787289643616 [ 6643.967913] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6643.974020] LustreError: 221641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6644.209958] Lustre: server umount lustre-MDT0001 complete [ 6655.663155] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6658.526229] Key type lgssc unregistered [ 6658.835082] LNet: 223877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6658.845928] LNetError: 223877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6659.877674] LNet: Removed LNI 192.168.203.128@tcp [ 6660.835460] Key type .llcrypt unregistered [ 6660.838889] Key type ._llcrypt unregistered [ 6706.060363] Key type ._llcrypt registered [ 6706.062337] Key type .llcrypt registered [ 6706.250081] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 6708.282704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6708.601665] alg: No test for adler32 (adler32-zlib) [ 6710.072313] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 6710.280068] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 6711.991156] Key type lgssc registered [ 6713.610467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6723.544463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6723.565303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6724.923327] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6724.988506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6725.202178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6729.753846] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6739.667591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.758503] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6739.777856] Lustre: 226083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6740.031097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6745.414502] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6753.051531] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6759.641244] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6767.672308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6768.028960] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6768.392322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6774.995348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:577) [ 6776.356283] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 6784.707910] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6787.519481] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6793.934585] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6802.497812] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6811.404520] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6825.150895] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6825.154854] Lustre: Skipped 1 previous similar message [ 6847.646751] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6847.661414] Lustre: Skipped 2 previous similar messages [ 6884.588383] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6884.594953] Lustre: Skipped 4 previous similar messages [ 6954.470612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6954.473311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6954.504835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6954.516534] Lustre: Skipped 1 previous similar message [ 6955.866783] Lustre: server umount lustre-OST0000 complete [ 6960.096328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6960.108325] LustreError: Skipped 1 previous similar message [ 6960.114591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6960.128764] Lustre: Skipped 1 previous similar message [ 6960.132761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6964.705896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6964.707791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6964.750047] Lustre: Skipped 1 previous similar message [ 6966.231751] Lustre: server umount lustre-MDT0000 complete [ 6970.425649] LustreError: 226076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846852 with bad export cookie 3911564009808647097 [ 6970.433514] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6970.444892] LustreError: 226076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6970.833815] Lustre: server umount lustre-MDT0001 complete [ 6983.741595] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 6987.582545] Key type lgssc unregistered [ 6987.956402] LNet: 230889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6987.966364] LNetError: 230889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6989.042332] LNet: Removed LNI 192.168.203.128@tcp [ 6990.134210] Key type .llcrypt unregistered [ 6990.139612] Key type ._llcrypt unregistered [ 7022.635918] Key type ._llcrypt registered [ 7022.638577] Key type .llcrypt registered [ 7022.738590] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 7025.022250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7025.069747] alg: No test for adler32 (adler32-zlib) [ 7026.450275] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 7026.860544] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 7028.679150] Key type lgssc registered [ 7030.199512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7039.895642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7039.925175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7041.425654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7045.784996] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7051.750580] LustreError: 232304:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.771968] LustreError: 232304:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7054.577784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7055.030632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7059.642867] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7067.997067] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7074.217497] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7081.478856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7081.907325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7087.024771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:609) [ 7087.841727] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_default_debug -1 all [ 7095.130931] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7112.676543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7112.694829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.710896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7114.733443] LustreError: 234128:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7114.746374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7114.774257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7114.844175] Lustre: server umount lustre-OST0000 complete [ 7119.843326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.864289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7119.881612] Lustre: Skipped 1 previous similar message [ 7124.966071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7124.978282] Lustre: Skipped 1 previous similar message [ 7125.324546] Lustre: server umount lustre-MDT0000 complete [ 7129.512776] LustreError: 232285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847011 with bad export cookie 13205623904609988512 [ 7129.544616] LustreError: MGC192.168.203.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7130.210994] Lustre: server umount lustre-MDT0001 complete [ 7144.307248] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 7147.953304] Key type lgssc unregistered [ 7148.557769] LNet: 236003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7148.589755] LNetError: 236003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7148.648738] LNet: Removed LNI 192.168.203.128@tcp [ 7149.961385] Key type .llcrypt unregistered [ 7149.967435] Key type ._llcrypt unregistered [ 7178.912541] Key type ._llcrypt registered [ 7178.915215] Key type .llcrypt registered [ 7179.008229] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 7180.179838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7180.320487] alg: No test for adler32 (adler32-zlib) [ 7181.665950] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 7181.960175] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 7183.663442] Key type lgssc registered [ 7185.246201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7457.099910] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 7460.436810] Key type lgssc unregistered [ 7460.850715] LNet: 238030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7460.856682] LNetError: 238030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7460.895260] LNet: Removed LNI 192.168.203.128@tcp [ 7461.858390] Key type .llcrypt unregistered [ 7461.864241] Key type ._llcrypt unregistered [ 7490.539693] Key type ._llcrypt registered [ 7490.544671] Key type .llcrypt registered [ 7490.641844] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing set_hostid [ 7509.558861] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing load_modules_local [ 7510.803367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7510.860590] alg: No test for adler32 (adler32-zlib) [ 7511.975977] Lustre: Lustre: Build Version: 2.17.57_95_g5b39979 [ 7512.271552] LNet: Added LNI 192.168.203.128@tcp [8/256/0/180] [ 7513.975164] Key type lgssc registered [ 7514.988437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7555.119349] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7286 sec ========== 12:17:14 (1787847434) [ 7556.989707] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:17:17 (1787847437) === [ 7562.339384] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:17:22 (1787847442) === [ 7605.396378] Lustre: DEBUG MARKER: oleg328-server.virtnet: executing unload_modules_local [ 7607.950403] Key type lgssc unregistered [ 7608.249718] LNet: 243272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7608.258462] LNetError: 243272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7608.270721] LNet: Removed LNI 192.168.203.128@tcp [ 7609.145418] Key type .llcrypt unregistered [ 7609.147482] Key type ._llcrypt unregistered