[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 452245298 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003413] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009023] pid_max: default: 32768 minimum: 301 [ 0.010178] LSM: Security Framework initializing [ 0.011093] Yama: becoming mindful. [ 0.012050] SELinux: Initializing. [ 0.014084] *** VALIDATE selinux *** [ 0.023043] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028037] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030197] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032051] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033158] *** VALIDATE tmpfs *** [ 0.035369] *** VALIDATE proc *** [ 0.037086] *** VALIDATE cgroup *** [ 0.038011] *** VALIDATE cgroup2 *** [ 0.040069] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043039] Spectre V2 : User space: Vulnerable [ 0.044012] Speculative Store Bypass: Vulnerable [ 0.047235] debug: unmapping init [mem 0xffffffffb4459000-0xffffffffb4460fff] [ 0.050000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050683] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051021] ... version: 2 [ 0.052012] ... bit width: 48 [ 0.053015] ... generic registers: 4 [ 0.054016] ... value mask: 0000ffffffffffff [ 0.055017] ... max period: 00007fffffffffff [ 0.056018] ... fixed-purpose events: 3 [ 0.057019] ... event mask: 000000070000000f [ 0.059340] rcu: Hierarchical SRCU implementation. [ 0.061827] smp: Bringing up secondary CPUs ... [ 0.062753] x86: Booting SMP configuration: [ 0.063042] .... node #0, CPUs: #1 #2 #3 [ 0.070035] smp: Brought up 1 node, 4 CPUs [ 0.072016] smpboot: Max logical packages: 1 [ 0.073026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.239060] node 0 deferred pages initialised in 164ms [ 0.244134] devtmpfs: initialized [ 0.245354] x86/mm: Memory block size: 128MB [ 0.248713] gcov: version magic: 0x41383552 [ 0.251340] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.254092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.256468] pinctrl core: initialized pinctrl subsystem [ 0.258212] [ 0.258878] ************************************************************* [ 0.261019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264020] ** ** [ 0.266016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.268014] ** ** [ 0.270013] ** This means that this kernel is built to expose internal ** [ 0.272011] ** IOMMU data structures, which may compromise security on ** [ 0.273013] ** your system. ** [ 0.276016] ** ** [ 0.278013] ** If you see this message and you are not debugging the ** [ 0.280018] ** kernel, report this immediately to your vendor! ** [ 0.281013] ** ** [ 0.283016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285017] ************************************************************* [ 0.287873] NET: Registered protocol family 16 [ 0.290490] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.293090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.296166] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.301039] cpuidle: using governor menu [ 0.303000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.307784] PCI: Using configuration type 1 for base access [ 0.310172] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.319159] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.323049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.329086] cryptd: max_cpu_qlen set to 1000 [ 0.332334] ACPI: Added _OSI(Module Device) [ 0.334014] ACPI: Added _OSI(Processor Device) [ 0.336015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.338016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.343000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.349367] ACPI: Interpreter enabled [ 0.350058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.352013] ACPI: Using IOAPIC for interrupt routing [ 0.353114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.356459] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.365844] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.369093] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.426039] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.430132] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.435622] acpiphp: Slot [2] registered [ 0.437154] acpiphp: Slot [5] registered [ 0.438110] acpiphp: Slot [6] registered [ 0.439006] acpiphp: Slot [7] registered [ 0.440117] acpiphp: Slot [8] registered [ 0.441115] acpiphp: Slot [9] registered [ 0.442110] acpiphp: Slot [10] registered [ 0.443137] acpiphp: Slot [3] registered [ 0.445099] acpiphp: Slot [4] registered [ 0.446092] acpiphp: Slot [11] registered [ 0.447101] acpiphp: Slot [12] registered [ 0.448112] acpiphp: Slot [13] registered [ 0.450352] acpiphp: Slot [14] registered [ 0.451377] acpiphp: Slot [15] registered [ 0.453143] acpiphp: Slot [16] registered [ 0.455166] acpiphp: Slot [17] registered [ 0.456092] acpiphp: Slot [18] registered [ 0.458114] acpiphp: Slot [19] registered [ 0.459187] acpiphp: Slot [20] registered [ 0.460131] acpiphp: Slot [21] registered [ 0.462247] acpiphp: Slot [22] registered [ 0.465172] acpiphp: Slot [23] registered [ 0.466081] acpiphp: Slot [24] registered [ 0.467071] acpiphp: Slot [25] registered [ 0.468079] acpiphp: Slot [26] registered [ 0.469078] acpiphp: Slot [27] registered [ 0.470120] acpiphp: Slot [28] registered [ 0.472093] acpiphp: Slot [29] registered [ 0.473083] acpiphp: Slot [30] registered [ 0.474084] acpiphp: Slot [31] registered [ 0.475097] PCI host bridge to bus 0000:00 [ 0.476019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.478034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.481034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.484028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.487030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.490037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.493200] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.496277] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.499791] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.510018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.515064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.518086] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.521024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.524022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.526413] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.529746] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.533060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.535789] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.540015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.557071] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.566017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.572234] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.582020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.590022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.609025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.628519] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.642017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.649016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.679016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.689279] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.701022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.711023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.763021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.786696] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.795019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.805038] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.831028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.846562] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.854019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.866021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.892021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.903520] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.915015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.929094] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.959022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.977558] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.980459] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.983556] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.987445] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.989317] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.994305] iommu: Default domain type: Passthrough [ 1.003647] SCSI subsystem initialized [ 1.007582] ACPI: bus type USB registered [ 1.012232] usbcore: registered new interface driver usbfs [ 1.016238] usbcore: registered new interface driver hub [ 1.020142] usbcore: registered new device driver usb [ 1.023334] pps_core: LinuxPPS API ver. 1 registered [ 1.024015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.028088] PTP clock support registered [ 1.031053] EDAC MC: Ver: 3.0.0 [ 1.033234] PCI: Using ACPI for IRQ routing [ 1.034987] NetLabel: Initializing [ 1.036018] NetLabel: domain hash size = 128 [ 1.039014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.041134] NetLabel: unlabeled traffic allowed by default [ 1.045012] vgaarb: loaded [ 1.047048] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.049012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.061604] clocksource: Switched to clocksource kvm-clock [ 1.211247] VFS: Disk quotas dquot_6.6.0 [ 1.212967] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.215722] *** VALIDATE ramfs *** [ 1.217069] *** VALIDATE hugetlbfs *** [ 1.218816] pnp: PnP ACPI init [ 1.221172] pnp: PnP ACPI: found 6 devices [ 1.237880] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.241395] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.243818] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.246204] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.249033] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.251576] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.254142] NET: Registered protocol family 2 [ 1.256594] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.262419] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.266696] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.272498] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.276239] TCP: Hash tables configured (established 65536 bind 65536) [ 1.279653] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.283289] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.286532] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.289899] NET: Registered protocol family 1 [ 1.292792] RPC: Registered named UNIX socket transport module. [ 1.295487] RPC: Registered udp transport module. [ 1.297548] RPC: Registered tcp transport module. [ 1.299441] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.302112] NET: Registered protocol family 44 [ 1.303971] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.306317] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.308698] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.310988] PCI: CLS 0 bytes, default 64 [ 1.312529] Unpacking initramfs... [ 3.104801] debug: unmapping init [mem 0xffff9282fcc54000-0xffff9282fffbffff] [ 3.111440] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.113092] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.116426] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.186778] Initialise system trusted keyrings [ 4.190389] Key type blacklist registered [ 4.193387] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.202465] zbud: loaded [ 4.205521] *** VALIDATE nfs *** [ 4.206966] *** VALIDATE nfs4 *** [ 4.208608] pstore: using deflate compression [ 4.212372] Platform Keyring initialized [ 4.341453] NET: Registered protocol family 38 [ 4.343770] Key type asymmetric registered [ 4.345893] Asymmetric key parser 'x509' registered [ 4.351887] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.360641] io scheduler mq-deadline registered [ 4.364305] io scheduler kyber registered [ 4.369798] io scheduler bfq registered [ 4.373912] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.381790] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.393380] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.398431] ACPI: Power Button [PWRF] [ 4.406699] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.414817] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.435856] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.447819] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.466819] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.502928] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.557334] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.568129] Non-volatile memory driver v1.3 [ 4.570081] Linux agpgart interface v0.103 [ 4.673381] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 4.684364] vda: detected capacity change from 0 to 76787712 [ 4.723070] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.732090] vdb: detected capacity change from 0 to 1073741824 [ 4.768654] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.776681] vdc: detected capacity change from 0 to 2621440000 [ 4.817100] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.825681] vdd: detected capacity change from 0 to 2621440000 [ 4.851632] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.859444] vde: detected capacity change from 0 to 4294967296 [ 4.891182] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.897173] vdf: detected capacity change from 0 to 4294967296 [ 4.908763] libphy: Fixed MDIO Bus: probed [ 4.924354] usbcore: registered new interface driver usbserial_generic [ 4.932087] usbserial: USB Serial support registered for generic [ 4.939304] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.949581] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.953458] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.961340] mousedev: PS/2 mouse device common for all mice [ 4.969625] rtc_cmos 00:05: RTC can wake from S4 [ 4.979093] rtc_cmos 00:05: registered as rtc0 [ 4.981669] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.986688] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.988751] intel_pstate: CPU model not supported [ 4.997405] hid: raw HID events driver (C) Jiri Kosina [ 5.018341] usbcore: registered new interface driver usbhid [ 5.020441] usbhid: USB HID core driver [ 5.027941] drop_monitor: Initializing network drop monitor service [ 5.037573] Initializing XFRM netlink socket [ 5.038391] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.056377] NET: Registered protocol family 10 [ 5.068229] Segment Routing with IPv6 [ 5.076139] NET: Registered protocol family 17 [ 5.078766] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.089260] mpls_gso: MPLS GSO support [ 5.108513] RAS: Correctable Errors collector initialized. [ 5.111717] AVX version of gcm_enc/dec engaged. [ 5.114637] AES CTR mode by8 optimization enabled [ 5.219634] sched_clock: Marking stable (5219617373, 0)->(6134152685, -914535312) [ 5.223389] registered taskstats version 1 [ 5.225865] Loading compiled-in X.509 certificates [ 5.228249] zswap: loaded using pool lzo/zbud [ 5.254435] Key type big_key registered [ 5.270221] Key type encrypted registered [ 5.271784] ima: No TPM chip found, activating TPM-bypass! [ 5.275637] ima: Allocated hash algorithm: sha1 [ 5.277777] ima: No architecture policies found [ 5.279652] evm: Initialising EVM extended attributes: [ 5.281723] evm: security.selinux [ 5.284513] evm: security.ima [ 5.285878] evm: security.capability [ 5.287919] evm: HMAC attrs: 0x1 [ 5.291244] rtc_cmos 00:05: setting system clock to 2026-09-17 22:11:09 UTC (1789683069) [ 5.317713] debug: unmapping init [mem 0xffffffffb5403000-0xffffffffb55fffff] [ 5.321674] debug: unmapping init [mem 0xffffffffb4182000-0xffffffffb4458fff] [ 5.333122] Write protecting the kernel read-only data: 28672k [ 5.340738] debug: unmapping init [mem 0xffffffffb2803000-0xffffffffb29fffff] [ 5.346031] debug: unmapping init [mem 0xffffffffb3114000-0xffffffffb31fffff] [ 5.385276] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.395116] systemd[1]: Detected virtualization kvm. [ 5.396990] systemd[1]: Detected architecture x86-64. [ 5.398871] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.424606] systemd[1]: No hostname configured. [ 5.426475] systemd[1]: Set hostname to . [ 5.429507] random: systemd: uninitialized urandom read (16 bytes read) [ 5.432619] systemd[1]: Initializing machine ID from random generator. [ 5.705805] random: systemd: uninitialized urandom read (16 bytes read) [ 5.708922] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.723404] random: systemd: uninitialized urandom read (16 bytes read) [ 5.726557] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.748820] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.088945] device-mapper: uevent: version 1.0.3 [ 7.091284] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook...[ 7.715534] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.851839] virtio_net virtio0 ens2: renamed from eth0 [ 9.241643] scsi host0: ata_piix [ 9.430665] scsi host1: ata_piix [ 9.431881] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.451133] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.814443] random: crng init done [ 13.815528] random: 7 urandom warning(s) missed due to ratelimiting [ 17.380293] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.783833] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ 20.328021] hrtimer: interrupt took 5013143 ns [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.815257] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.686660] SELinux: Disabled at runtime. [ 23.799350] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.807408] systemd[1]: Detected virtualization kvm. [ 23.809230] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.471415] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.480530] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.505436] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.515132] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.520408] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.542984] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.568576] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 25.911335] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 27.161142] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.298093] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.425354] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.081514] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.354276] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 33.582482] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 34.344452] NFS: Registering the id_resolver key type [ 34.347178] Key type id_resolver registered [ 34.349237] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg309-server login: [ 82.426675] libcfs: loading out-of-tree module taints kernel. [ 82.462561] Key type ._llcrypt registered [ 82.464495] Key type .llcrypt registered [ 82.530339] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 96.491693] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 97.480780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 97.494571] alg: No test for adler32 (adler32-zlib) [ 98.665933] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 99.263279] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 101.008398] Key type lgssc registered [ 102.137653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.502094] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 141.293929] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 150.397463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 150.420817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.605708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 151.641372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 151.703369] Lustre: lustre-MDT0000: new disk, initializing [ 151.766373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 151.797590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 154.923674] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.615266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.695452] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.726155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.729640] Lustre: Skipped 1 previous similar message [ 164.791568] Lustre: lustre-MDT0001: new disk, initializing [ 164.835207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.857289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.866655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 167.634459] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.032881] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 177.788293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 178.132915] Lustre: lustre-OST0000: new disk, initializing [ 178.142612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 178.154595] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 178.244139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.435029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 183.449273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 183.605237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 187.461942] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.404258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.513994] Lustre: lustre-OST0001: new disk, initializing [ 206.517345] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 206.521643] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.575989] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.073499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 214.100039] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 214.171239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 215.871337] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.103487] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 243.910437] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.382767] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing check_logdir /tmp/testlogs/ [ 258.615142] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing yml_node [ 263.918806] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 266.747205] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 269.979316] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 271.738751] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 17 18:15:35 EDT 2026 [ 291.643334] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 293.986517] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 311.293514] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.302237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.306329] Lustre: Skipped 3 previous similar messages [ 311.317869] Lustre: Skipped 3 previous similar messages [ 315.925524] Lustre: server umount lustre-MDT0000 complete [ 319.703095] LustreError: 9528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683383 with bad export cookie 7599160312464066542 [ 319.709324] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.714077] LustreError: 9528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 321.512319] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.516586] LustreError: 6525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.516937] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.530840] Lustre: Skipped 1 previous similar message [ 321.576952] LustreError: 6525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 324.513603] LustreError: 6527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.517063] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.539574] LustreError: 6527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 324.563205] Lustre: Skipped 2 previous similar messages [ 329.633292] LustreError: 6527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.634256] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.658181] LustreError: 6527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 329.681609] Lustre: Skipped 1 previous similar message [ 334.304665] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 334.650572] Lustre: server umount lustre-MDT0001 complete [ 339.278955] Lustre: server umount lustre-OST0000 complete [ 343.043732] Lustre: server umount lustre-OST0001 complete [ 350.297851] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 361.172313] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 400.512135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.912463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 400.953153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.049331] Lustre: lustre-MDT0000: new disk, initializing [ 401.154067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.179363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.323835] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 417.479212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.640857] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.770568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.780972] Lustre: Skipped 1 previous similar message [ 417.934891] Lustre: lustre-MDT0001: new disk, initializing [ 418.068972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 418.102255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 418.118132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 422.701955] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 433.174982] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 439.112917] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 446.225195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 446.629423] Lustre: lustre-OST0000: new disk, initializing [ 446.638701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 446.645628] Lustre: 17056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 446.736899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.149896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 448.171020] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 448.331383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 453.740529] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 463.162096] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 468.298927] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 468.565066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 472.995807] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 473.190785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 479.203628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 479.217997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.225228] LustreError: Skipped 1 previous similar message [ 479.226364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 479.277531] Lustre: Skipped 1 previous similar message [ 481.247402] Lustre: server umount lustre-OST0000 complete [ 489.451091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.474647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 489.484271] Lustre: Skipped 2 previous similar messages [ 491.235103] Lustre: server umount lustre-MDT0000 complete [ 494.561958] LustreError: 15099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.591887] LustreError: 15099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 494.795111] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683559 with bad export cookie 7599160312464068068 [ 494.798881] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 494.803778] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 495.286726] Lustre: server umount lustre-MDT0001 complete [ 499.164679] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 18:19:22 (1789683562) [ 506.734909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 507.200242] LustreError: 19323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.323537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.228792] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 517.604534] LustreError: 19324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.640290] LustreError: 19324:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 523.594037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.112037] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 539.974740] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 546.830495] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 554.292940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 562.322646] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 572.681349] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 587.752401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 587.752445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.763314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 587.796225] Lustre: Skipped 2 previous similar messages [ 592.505529] Lustre: server umount lustre-OST0000 complete [ 597.987277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.051288] Lustre: server umount lustre-MDT0000 complete [ 603.108867] LustreError: 19318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.136102] LustreError: 19318:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 607.816295] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683672 with bad export cookie 7599160312464068957 [ 607.821848] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 607.827363] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 608.705649] Lustre: server umount lustre-MDT0001 complete [ 624.416777] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 628.536272] Key type lgssc unregistered [ 629.072896] LNet: 22880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.085278] LNetError: 22880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.119918] LNet: Removed LNI 192.168.203.109@tcp [ 631.849555] Key type .llcrypt unregistered [ 631.857077] Key type ._llcrypt unregistered [ 661.598644] Key type ._llcrypt registered [ 661.601352] Key type .llcrypt registered [ 661.764851] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 663.359836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 663.522905] alg: No test for adler32 (adler32-zlib) [ 664.744818] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 665.226365] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 666.920212] Key type lgssc registered [ 668.432457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 677.793263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 677.829528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.341561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.416270] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 689.635469] LustreError: 24294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.658647] LustreError: 24294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 694.758740] LustreError: 24293:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 696.873665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.380250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 703.044856] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 711.987692] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 719.640586] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 728.022670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 728.597281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 735.636154] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 744.074072] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 749.042679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 759.277939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.288713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 759.305236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 759.320201] Lustre: Skipped 1 previous similar message [ 764.388618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 764.398673] Lustre: Skipped 1 previous similar message [ 764.901337] Lustre: server umount lustre-OST0000 complete [ 769.006482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 769.020531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.029322] Lustre: Skipped 1 previous similar message [ 769.033868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.629187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.641741] Lustre: Skipped 3 previous similar messages [ 775.096783] Lustre: server umount lustre-MDT0000 complete [ 778.853865] LustreError: 26139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683843 with bad export cookie 15105767762771607328 [ 778.856842] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 778.871437] LustreError: 26139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 779.489474] Lustre: server umount lustre-MDT0001 complete [ 791.624474] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 794.577910] Key type lgssc unregistered [ 794.964628] LNet: 28268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.985343] LNetError: 28268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.008456] LNet: Removed LNI 192.168.203.109@tcp [ 795.895780] Key type .llcrypt unregistered [ 795.897478] Key type ._llcrypt unregistered [ 821.755236] Key type ._llcrypt registered [ 821.757846] Key type .llcrypt registered [ 821.935052] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 823.782913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 823.860812] alg: No test for adler32 (adler32-zlib) [ 825.148585] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 825.426881] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 827.096232] Key type lgssc registered [ 828.780381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 838.289682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 838.350237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.018747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.061098] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 850.402415] LustreError: 29678:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.460851] LustreError: 29678:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 855.341422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.526900] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 855.838215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 861.743939] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 871.570970] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 878.637881] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 895.250505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 895.918323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.112374] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 909.139486] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 910.266492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 920.546475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 920.557450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.561245] LustreError: Skipped 1 previous similar message [ 920.564730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 920.591860] Lustre: Skipped 1 previous similar message [ 925.666682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.682486] Lustre: Skipped 2 previous similar messages [ 926.360900] Lustre: server umount lustre-OST0000 complete [ 930.790449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.802214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 930.816922] Lustre: Skipped 1 previous similar message [ 930.833216] Lustre: Skipped 1 previous similar message [ 935.905917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 935.915922] Lustre: Skipped 1 previous similar message [ 936.792404] Lustre: server umount lustre-MDT0000 complete [ 940.832665] LustreError: 29657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684005 with bad export cookie 8034355819495754074 [ 940.856933] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 941.347711] Lustre: server umount lustre-MDT0001 complete [ 955.935248] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 959.843102] Key type lgssc unregistered [ 960.310198] LNet: 33560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.371120] LNetError: 33560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.401659] LNet: Removed LNI 192.168.203.109@tcp [ 961.974440] Key type .llcrypt unregistered [ 961.981095] Key type ._llcrypt unregistered [ 988.894982] Key type ._llcrypt registered [ 988.900091] Key type .llcrypt registered [ 989.038967] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 991.379592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 991.438163] alg: No test for adler32 (adler32-zlib) [ 992.758301] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 993.091071] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 994.816239] Key type lgssc registered [ 997.072184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1008.102172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1008.131473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.782719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.713687] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1019.364219] LustreError: 34971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.393549] LustreError: 34971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1024.493153] LustreError: 34970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.691434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.622781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1033.645282] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1041.527638] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1050.068949] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1059.523537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1060.076425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.552284] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1077.261976] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1079.562626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1093.096559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1093.104224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.126738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1094.117850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1094.122359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1094.134135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1095.140370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1095.155650] Lustre: Skipped 1 previous similar message [ 1098.544629] Lustre: server umount lustre-OST0000 complete [ 1105.378246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.378715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1105.399329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1105.416366] Lustre: Skipped 1 previous similar message [ 1108.097516] Lustre: server umount lustre-MDT0000 complete [ 1110.503549] LustreError: 35711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.523339] LustreError: 35711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1111.277538] LustreError: 34950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684175 with bad export cookie 4583556361004311516 [ 1111.291648] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1111.295106] LustreError: 34950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1111.584369] Lustre: server umount lustre-MDT0001 complete [ 1120.913549] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1123.639906] Key type lgssc unregistered [ 1123.942772] LNet: 38526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.949257] LNetError: 38526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.963653] LNet: Removed LNI 192.168.203.109@tcp [ 1124.691466] Key type .llcrypt unregistered [ 1124.694840] Key type ._llcrypt unregistered [ 1151.049733] Key type ._llcrypt registered [ 1151.054095] Key type .llcrypt registered [ 1151.152974] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1152.939125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1153.247871] alg: No test for adler32 (adler32-zlib) [ 1154.347824] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1154.655815] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1156.344665] Key type lgssc registered [ 1157.349312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1164.959405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1165.022253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.700747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.437394] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1176.040451] LustreError: 39936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.085558] LustreError: 39936:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1179.796465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.259220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1184.690883] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1191.735425] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1198.409923] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1205.804587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1206.243022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.390596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1211.798112] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1219.316394] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1230.831475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.839961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1230.849876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.257244] Lustre: lustre-OST0000: Not available for connect from 192.168.203.9@tcp (stopping) [ 1232.268027] Lustre: Skipped 1 previous similar message [ 1236.034585] Lustre: server umount lustre-OST0000 complete [ 1256.424518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.441625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.442363] Lustre: Skipped 2 previous similar messages [ 1256.465880] Lustre: Skipped 1 previous similar message [ 1261.541680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.547825] Lustre: Skipped 1 previous similar message [ 1261.756172] Lustre: server umount lustre-MDT0000 complete [ 1266.149323] LustreError: 39918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684330 with bad export cookie 1298303356834653098 [ 1266.156125] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1266.158330] LustreError: 39918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1272.499895] Lustre: server umount lustre-MDT0001 complete [ 1284.909306] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1288.027681] Key type lgssc unregistered [ 1288.417162] LNet: 43643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.425944] LNetError: 43643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.458644] LNet: Removed LNI 192.168.203.109@tcp [ 1289.361157] Key type .llcrypt unregistered [ 1289.363057] Key type ._llcrypt unregistered [ 1314.127284] Key type ._llcrypt registered [ 1314.135149] Key type .llcrypt registered [ 1314.235619] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1315.485446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1315.597302] alg: No test for adler32 (adler32-zlib) [ 1316.836424] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1317.110095] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1318.832278] Key type lgssc registered [ 1320.245216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1329.209686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1329.233727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.935868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.151390] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1341.410761] LustreError: 45052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.433372] LustreError: 45052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1345.373147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.702072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1350.282581] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1358.258684] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1364.746447] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1372.760276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1373.146780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.176262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1380.158543] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1389.009782] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1395.568665] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) [ 1395.687602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1395.708813] Lustre: Skipped 1 previous similar message [ 1400.673213] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) [ 1400.686245] Lustre: Skipped 3 previous similar messages [ 1401.024356] Lustre: server umount lustre-MDT0000 complete [ 1404.524865] LustreError: 46895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684468 with bad export cookie 14001536695674699709 [ 1404.531557] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1404.533021] LustreError: 46895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1405.787663] LustreError: 45046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.827110] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.9@tcp (stopping) [ 1405.835587] Lustre: Skipped 3 previous similar messages [ 1405.925515] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.939871] Lustre: Skipped 1 previous similar message [ 1409.959119] LustreError: 45046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.959144] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1409.972527] LustreError: 45046:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1409.978291] Lustre: Skipped 1 previous similar message [ 1410.957587] Lustre: server umount lustre-MDT0001 complete [ 1431.283663] Lustre: server umount lustre-OST0000 complete [ 1446.073216] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1449.057149] Key type lgssc unregistered [ 1449.482131] LNet: 48913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.498552] LNetError: 48913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.519373] LNet: Removed LNI 192.168.203.109@tcp [ 1450.570332] Key type .llcrypt unregistered [ 1450.573540] Key type ._llcrypt unregistered [ 1472.917388] Key type ._llcrypt registered [ 1472.919302] Key type .llcrypt registered [ 1473.018896] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1473.942285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1474.074139] alg: No test for adler32 (adler32-zlib) [ 1475.097507] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1475.316454] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1476.984219] Key type lgssc registered [ 1477.918617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1484.318120] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1484.339709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1499.943670] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1505.056392] LustreError: 50291:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 1531.744232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1536.732921] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1543.114670] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1709.066312] Lustre: server umount lustre-OST0000 complete [ 1721.342915] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1723.426616] Key type lgssc unregistered [ 1723.668777] LNet: 51963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.675812] LNetError: 51963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.685875] LNet: Removed LNI 192.168.203.109@tcp [ 1724.353127] Key type .llcrypt unregistered [ 1724.356693] Key type ._llcrypt unregistered [ 1741.716590] Key type ._llcrypt registered [ 1741.718724] Key type .llcrypt registered [ 1741.797223] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1742.755754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1742.787203] alg: No test for adler32 (adler32-zlib) [ 1743.806607] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1744.000711] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1745.664207] Key type lgssc registered [ 1746.549447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1752.448499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1752.473315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.880589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1756.668523] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1762.232816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1762.528031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1765.672467] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1771.108863] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1774.973100] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1779.777505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1780.034784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.181349] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1789.469461] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1792.485212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1792.491210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.499189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1793.227474] Lustre: server umount lustre-OST0000 complete [ 1797.606334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.609233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.624843] Lustre: Skipped 1 previous similar message [ 1797.632037] Lustre: Skipped 1 previous similar message [ 1802.721621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.727830] Lustre: Skipped 1 previous similar message [ 1807.842868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1807.848321] Lustre: Skipped 1 previous similar message [ 1810.401441] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1810.600856] Lustre: server umount lustre-MDT0000 complete [ 1811.951152] LustreError: 53352:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1811.969935] LustreError: 53352:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1813.418628] LustreError: 53333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684877 with bad export cookie 5100741991587821462 [ 1813.420496] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1813.431304] LustreError: 53333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1813.698664] Lustre: server umount lustre-MDT0001 complete [ 1820.743597] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1822.535106] Key type lgssc unregistered [ 1822.743523] LNet: 56881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.749653] LNetError: 56881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.769195] LNet: Removed LNI 192.168.203.109@tcp [ 1823.408492] Key type .llcrypt unregistered [ 1823.410158] Key type ._llcrypt unregistered [ 1838.264160] Key type ._llcrypt registered [ 1838.266295] Key type .llcrypt registered [ 1838.334318] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1839.008897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.018769] alg: No test for adler32 (adler32-zlib) [ 1839.960952] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1840.140432] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1841.776369] Key type lgssc registered [ 1842.431782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1847.275904] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1847.301039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1862.816404] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1867.936337] LustreError: 58241:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 1894.624872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.358943] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1902.954439] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1907.145156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.489662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.544801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1910.655550] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1914.855151] LustreError: 58267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.870250] LustreError: 58267:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 1915.923773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.162251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1918.654394] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1924.059298] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1927.831277] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1931.748981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.751054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1931.758317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1931.768013] Lustre: Skipped 1 previous similar message [ 1935.126866] Lustre: server umount lustre-OST0000 complete [ 1957.347057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1957.347225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.354202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.360160] Lustre: Skipped 2 previous similar messages [ 1958.585839] Lustre: server umount lustre-MDT0000 complete [ 1960.727590] LustreError: 58249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685024 with bad export cookie 6350689494478585274 [ 1960.728191] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1960.736332] LustreError: 58249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1975.264740] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1975.478521] Lustre: server umount lustre-MDT0001 complete [ 1982.251304] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1984.017296] Key type lgssc unregistered [ 1984.230750] LNet: 61980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.236140] LNetError: 61980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.249517] LNet: Removed LNI 192.168.203.109@tcp [ 1984.814590] Key type .llcrypt unregistered [ 1984.816886] Key type ._llcrypt unregistered [ 1999.559817] Key type ._llcrypt registered [ 1999.561871] Key type .llcrypt registered [ 1999.644432] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2000.360661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2000.376155] alg: No test for adler32 (adler32-zlib) [ 2001.324592] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2001.498559] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2003.120197] Key type lgssc registered [ 2003.741945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2008.244412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2008.257473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.545950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.373388] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2018.111095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.348293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2020.756882] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2025.155611] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2028.791220] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2032.652879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2032.853557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2033.897800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2035.927397] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2039.869429] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2058.731434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.732420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2058.735442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2058.746119] Lustre: Skipped 1 previous similar message [ 2061.162935] Lustre: server umount lustre-OST0000 complete [ 2063.842684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.862811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2063.870735] Lustre: Skipped 2 previous similar messages [ 2068.962099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2068.967032] Lustre: Skipped 1 previous similar message [ 2069.769703] Lustre: server umount lustre-MDT0000 complete [ 2071.863556] LustreError: 63347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685136 with bad export cookie 14412441403944869327 [ 2071.867502] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2071.871666] LustreError: 63347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2072.059211] Lustre: server umount lustre-MDT0001 complete [ 2078.378470] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2080.004440] Key type lgssc unregistered [ 2080.206736] LNet: 66917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.211531] LNetError: 66917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.222275] LNet: Removed LNI 192.168.203.109@tcp [ 2080.761272] Key type .llcrypt unregistered [ 2080.763588] Key type ._llcrypt unregistered [ 2094.751221] Key type ._llcrypt registered [ 2094.754119] Key type .llcrypt registered [ 2094.830318] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2095.276340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2095.286352] alg: No test for adler32 (adler32-zlib) [ 2096.186356] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2096.326932] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2097.936229] Key type lgssc registered [ 2098.641958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2103.908304] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 18:46:08 (1789685168) [ 2108.185967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2108.200922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.557294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2112.144387] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2117.074227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.298222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2120.031993] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2124.357505] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2127.612719] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2131.250922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2131.442195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.508663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2134.597088] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2138.714309] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2146.130527] Lustre: Failing over lustre-MDT0000 [ 2146.278422] Lustre: server umount lustre-MDT0000 complete [ 2147.812204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.817048] LustreError: 68591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.820487] Lustre: Skipped 2 previous similar messages [ 2147.832500] LustreError: 68591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2150.256925] LustreError: 68591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.268369] LustreError: 68591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2151.926296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.042833] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2152.156374] Lustre: *** cfs_fail_loc=135, val=0*** [ 2152.212952] LustreError: 71680:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2152.221573] LustreError: 71680:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.109@tcp: cfg command failed: rc = -2 [ 2152.231435] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2152.231435] [ 2152.245477] LustreError: MGC192.168.203.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2152.255168] LustreError: 71669:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2152.260187] LustreError: 71669:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2152.269678] LustreError: 71669:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2152.276315] LustreError: 71669:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2152.289668] Lustre: server umount lustre-MDT0000 complete [ 2152.291647] LustreError: 71669:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2152.930258] LustreError: 68595:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.949020] LustreError: 68595:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2155.371745] LustreError: 68592:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.376261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.515118] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2158.662868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.699763] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2160.500354] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2163.702900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2163.747123] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2163.817025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2165.006446] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2174.437234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2174.445930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.456711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2178.562768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2178.572165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.582720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2180.664479] Lustre: server umount lustre-OST0000 complete [ 2189.284111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.288319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2189.308129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.308137] Lustre: Skipped 2 previous similar messages [ 2191.096849] Lustre: server umount lustre-MDT0000 complete [ 2194.404569] LustreError: 69323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.426504] LustreError: 69323:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2195.137356] LustreError: 68577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685259 with bad export cookie 11172062217127013579 [ 2195.142281] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.146058] LustreError: 68577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2195.681849] Lustre: server umount lustre-MDT0001 complete [ 2208.572669] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2211.144542] Key type lgssc unregistered [ 2211.360494] LNet: 73457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.364874] LNetError: 73457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.379269] LNet: Removed LNI 192.168.203.109@tcp [ 2212.183635] Key type .llcrypt unregistered [ 2212.187360] Key type ._llcrypt unregistered [ 2231.473682] Key type ._llcrypt registered [ 2231.477959] Key type .llcrypt registered [ 2231.552929] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2232.452858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.533322] alg: No test for adler32 (adler32-zlib) [ 2233.720833] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2233.975557] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2235.648285] Key type lgssc registered [ 2236.807599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2243.383769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2243.409762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.821133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.457267] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2254.311153] LustreError: 74847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.342477] LustreError: 74847:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2255.156483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.424223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2258.924122] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2264.922088] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2269.253320] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2274.394267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2274.720178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.827430] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2285.831111] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2289.077488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2296.103614] Lustre: Failing over lustre-MDT0000 [ 2296.291604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2296.301185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.314104] LustreError: 74846:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.401455] Lustre: server umount lustre-MDT0000 complete [ 2297.709023] LustreError: 74843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.367267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.805411] LustreError: 74842:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.832415] LustreError: 74842:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2305.056727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.196562] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2305.394441] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2305.532621] LustreError: 77930:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2305.536439] LustreError: 77930:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.109@tcp: cfg command failed: rc = -9 [ 2305.549812] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2305.549812] [ 2305.563657] LustreError: MGC192.168.203.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2305.572757] LustreError: 77919:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2305.577035] LustreError: 77919:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2305.581681] LustreError: 77919:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2305.587702] LustreError: 77919:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2305.623254] Lustre: server umount lustre-MDT0000 complete [ 2305.625355] LustreError: 77919:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2307.925586] LustreError: 77332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.942796] LustreError: 77332:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2311.398447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.498164] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2311.778783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.841418] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2313.046467] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2315.775242] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2316.803810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2316.846800] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2316.921819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2324.847175] Lustre: server umount lustre-OST0000 complete [ 2332.130752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.133318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2332.141024] Lustre: Skipped 2 previous similar messages [ 2332.150355] Lustre: Skipped 1 previous similar message [ 2337.253451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.263453] Lustre: Skipped 1 previous similar message [ 2342.304889] Lustre: 74157:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789685390/real 1789685390] req@000000005d893eee x1876621051076864/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789685406 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2342.336559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.353539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2342.361215] Lustre: Skipped 1 previous similar message [ 2342.368198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2342.512965] Lustre: server umount lustre-MDT0000 complete [ 2343.392166] Lustre: 74158:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789685391/real 1789685391] req@0000000012f4fcba x1876621051077248/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789685407 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2345.462803] LustreError: 78961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685409 with bad export cookie 14527125654784186502 [ 2345.480611] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2345.481154] LustreError: 78961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2345.740495] Lustre: server umount lustre-MDT0001 complete [ 2353.962375] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2356.277735] Key type lgssc unregistered [ 2356.516644] LNet: 79712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.523368] LNetError: 79712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.536348] LNet: Removed LNI 192.168.203.109@tcp [ 2357.245779] Key type .llcrypt unregistered [ 2357.250677] Key type ._llcrypt unregistered [ 2367.512521] Key type ._llcrypt registered [ 2367.514943] Key type .llcrypt registered [ 2367.626304] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2367.870333] Key type .llcrypt unregistered [ 2367.873195] Key type ._llcrypt unregistered [ 2387.558245] Key type ._llcrypt registered [ 2387.560452] Key type .llcrypt registered [ 2387.652140] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2388.377988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.427601] alg: No test for adler32 (adler32-zlib) [ 2389.472609] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2389.747656] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2391.472360] Key type lgssc registered [ 2392.419991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2398.593508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2398.607167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.003081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.512958] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2410.033038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.326109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2413.982865] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2419.749231] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2423.897120] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2428.935992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2429.217752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.702254] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2440.438896] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2446.637930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2456.037637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2456.046435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2456.074282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2457.058724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2457.058908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2457.070429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.073881] Lustre: Skipped 1 previous similar message [ 2459.347971] Lustre: server umount lustre-OST0000 complete [ 2466.785667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2466.801269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.831934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2468.905226] Lustre: server umount lustre-MDT0000 complete [ 2472.284919] LustreError: 81568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685536 with bad export cookie 3744512492912491855 [ 2472.295887] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.671779] Lustre: server umount lustre-MDT0001 complete [ 2482.873509] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2485.070495] Key type lgssc unregistered [ 2485.342897] LNet: 85138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2485.353760] LNetError: 85138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2485.370329] LNet: Removed LNI 192.168.203.109@tcp [ 2486.131362] Key type .llcrypt unregistered [ 2486.136579] Key type ._llcrypt unregistered [ 2504.885441] Key type ._llcrypt registered [ 2504.887785] Key type .llcrypt registered [ 2504.984181] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2505.805074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2505.854120] alg: No test for adler32 (adler32-zlib) [ 2506.894284] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2507.104667] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2508.752358] Key type lgssc registered [ 2509.688842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2515.370982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2515.383473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.666291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.663165] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2526.178876] LustreError: 86525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.202401] LustreError: 86525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2527.395899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.640364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2531.274510] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2538.379443] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2544.857750] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2553.123440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2553.666936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.836090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2561.139906] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2570.581492] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2577.385573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2577.390499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.409268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2577.427416] Lustre: Skipped 1 previous similar message [ 2579.139906] Lustre: server umount lustre-OST0000 complete [ 2584.049681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2584.065421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.090412] Lustre: Skipped 1 previous similar message [ 2584.100478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.620017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.621901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.582480] Lustre: server umount lustre-MDT0000 complete [ 2592.743796] LustreError: 86526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.772487] LustreError: 86526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2595.955946] LustreError: 87259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685660 with bad export cookie 10727860075773986311 [ 2595.967248] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2596.502929] Lustre: server umount lustre-MDT0001 complete [ 2610.288139] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2614.514906] Key type lgssc unregistered [ 2615.018838] LNet: 90055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2615.030791] LNetError: 90055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2615.072919] LNet: Removed LNI 192.168.203.109@tcp [ 2616.496382] Key type .llcrypt unregistered [ 2616.498209] Key type ._llcrypt unregistered [ 2649.894159] Key type ._llcrypt registered [ 2649.896306] Key type .llcrypt registered [ 2650.062400] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2651.962743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2652.020104] alg: No test for adler32 (adler32-zlib) [ 2653.544516] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2654.002981] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2655.704263] Key type lgssc registered [ 2657.462639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2667.208878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2667.263062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.812717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2673.588142] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2678.262497] LustreError: 91466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.293814] LustreError: 91466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2682.944696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.287885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2690.044825] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2701.438544] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2709.075652] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2716.556961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2717.078522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2723.954318] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2732.652822] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2738.699722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2764.259768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2764.270825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2764.277597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2764.281872] Lustre: Skipped 1 previous similar message [ 2764.314717] LustreError: Skipped 1 previous similar message [ 2766.478448] Lustre: server umount lustre-OST0000 complete [ 2774.497575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2774.501687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2774.513465] Lustre: Skipped 1 previous similar message [ 2774.531194] Lustre: Skipped 2 previous similar messages [ 2778.527449] Lustre: server umount lustre-MDT0000 complete [ 2779.617442] LustreError: 91461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.655095] LustreError: 91461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2783.126193] LustreError: 91446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685847 with bad export cookie 13761483631168445099 [ 2783.138524] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2783.144293] LustreError: 91446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2783.558421] Lustre: server umount lustre-MDT0001 complete [ 2796.181282] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2799.600081] Key type lgssc unregistered [ 2800.012883] LNet: 95025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2800.024329] LNetError: 95025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2800.048374] LNet: Removed LNI 192.168.203.109@tcp [ 2801.004463] Key type .llcrypt unregistered [ 2801.007207] Key type ._llcrypt unregistered [ 2824.154484] Key type ._llcrypt registered [ 2824.157202] Key type .llcrypt registered [ 2824.282239] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2825.277173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2825.355662] alg: No test for adler32 (adler32-zlib) [ 2826.517866] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2826.798171] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2828.472238] Key type lgssc registered [ 2829.642413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2837.494588] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2837.535051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2853.154046] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2858.272640] LustreError: 96408:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 2885.090240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.300217] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2899.227285] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2905.592648] Lustre: server umount lustre-OST0000 complete [ 2914.201962] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:59:37 (1789685977) [ 2920.556397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.865320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2925.422497] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2926.057146] LustreError: 98046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.080159] LustreError: 98046:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2931.169809] LustreError: 98045:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2932.197530] LustreError: 98046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2934.193514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.679116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2939.289253] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2947.294668] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2953.542916] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2959.369057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2959.702545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2966.900142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2967.485537] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2975.195668] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2992.609864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2992.618079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.628226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2994.664993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2994.668235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2994.669565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.741399] Lustre: Skipped 1 previous similar message [ 2996.457149] Lustre: server umount lustre-OST0000 complete [ 3001.315108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3001.330386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.349730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3004.897296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3004.899689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.876438] Lustre: server umount lustre-MDT0000 complete [ 3010.018720] LustreError: 98042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3010.047463] LustreError: 98042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3011.465636] LustreError: 101001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686075 with bad export cookie 8512704094497438144 [ 3011.471325] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3011.472021] LustreError: 101001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3011.903224] Lustre: server umount lustre-MDT0001 complete [ 3025.066460] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 3028.160361] Key type lgssc unregistered [ 3028.402252] LNet: 101753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3028.422057] LNetError: 101753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3028.440628] LNet: Removed LNI 192.168.203.109@tcp [ 3029.396980] Key type .llcrypt unregistered [ 3029.399873] Key type ._llcrypt unregistered [ 3075.665708] Key type ._llcrypt registered [ 3075.669259] Key type .llcrypt registered [ 3075.846989] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3077.901241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3078.093073] alg: No test for adler32 (adler32-zlib) [ 3079.637150] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 3079.972563] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3081.659077] Key type lgssc registered [ 3083.508455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3094.030516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3094.062164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.881414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3100.303766] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3106.296842] LustreError: 103933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.319986] LustreError: 103933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3111.096353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.397770] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3111.756546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3119.421733] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3131.924836] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3141.186173] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3148.865121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.257649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3154.978026] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3162.238866] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3166.655129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3176.939161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3176.941568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.968535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3176.976524] Lustre: Skipped 1 previous similar message [ 3180.968881] Lustre: server umount lustre-OST0000 complete [ 3187.169957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3187.182074] Lustre: Skipped 1 previous similar message [ 3187.186808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3187.192629] Lustre: Skipped 1 previous similar message [ 3192.296639] LustreError: 106377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3192.338825] LustreError: 106377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3192.395436] Lustre: server umount lustre-MDT0000 complete [ 3195.530423] LustreError: 103913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686259 with bad export cookie 14976997864627394507 [ 3195.549020] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3195.549392] LustreError: 103913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3195.914895] Lustre: server umount lustre-MDT0001 complete [ 3204.736235] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 3207.060920] Key type lgssc unregistered [ 3207.293503] LNet: 107490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3207.301440] LNetError: 107490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3207.320617] LNet: Removed LNI 192.168.203.109@tcp [ 3208.226604] Key type .llcrypt unregistered [ 3208.240740] Key type ._llcrypt unregistered [ 3222.616423] Key type ._llcrypt registered [ 3222.618600] Key type .llcrypt registered [ 3222.687977] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3223.325057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3223.467833] alg: No test for adler32 (adler32-zlib) [ 3224.468393] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 3224.671398] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3226.328211] Key type lgssc registered [ 3227.367671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3233.766037] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3233.783794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3249.312391] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3254.434567] LustreError: 108708:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 3281.187656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.616236] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3300.434846] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3308.560080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.840869] LustreError: MGC192.168.203.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3308.850868] LustreError: 109734:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3308.863775] LustreError: 109734:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3308.869863] LustreError: 109734:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3308.938340] Lustre: server umount lustre-MDT0000 complete [ 3308.940647] LustreError: 109734:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3326.432368] Lustre: 108043:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789686374/real 1789686374] req@00000000580ba6fe x1876622089129344/t0(0) o400->MGC192.168.203.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789686390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3326.462818] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3326.477559] LustreError: 108720:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3347.085539] Lustre: server umount lustre-OST0000 complete [ 3358.421429] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3369.657815] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3404.632316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.909626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3404.976761] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3405.045093] Lustre: lustre-MDT0000: new disk, initializing [ 3405.095613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.109443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3409.559035] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3421.638500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.809176] Lustre: 112530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3421.946865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3421.958532] Lustre: Skipped 1 previous similar message [ 3422.061422] Lustre: lustre-MDT0001: new disk, initializing [ 3422.173630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3422.248714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3422.266724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3426.637312] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3436.615466] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3443.588627] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3452.727439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3453.085816] Lustre: lustre-OST0000: new disk, initializing [ 3453.092640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3453.109840] Lustre: 114496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3453.232454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3458.662859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3458.672240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3458.979763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3460.599348] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3474.182791] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3480.882671] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3481.446417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3487.393655] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3487.694348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3489.764496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3489.770082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3489.809842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3489.822554] Lustre: Skipped 1 previous similar message [ 3494.884284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3494.897735] Lustre: Skipped 1 previous similar message [ 3495.871850] Lustre: server umount lustre-OST0000 complete [ 3500.002564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3500.029533] Lustre: Skipped 1 previous similar message [ 3500.038859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3505.121433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3505.128988] Lustre: Skipped 2 previous similar messages [ 3510.243134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3510.253817] Lustre: Skipped 1 previous similar message [ 3514.336120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3514.597808] Lustre: server umount lustre-MDT0000 complete [ 3515.362169] LustreError: 112539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3515.383393] LustreError: 112539:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3518.500543] LustreError: 112521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686582 with bad export cookie 2536668690736977632 [ 3518.510915] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3518.992509] Lustre: server umount lustre-MDT0001 complete [ 3527.019106] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 19:09:50 (1789686590) [ 3530.418643] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3547.159967] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3556.714901] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3590.387579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.685404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3590.712828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3590.772129] Lustre: lustre-MDT0000: new disk, initializing [ 3590.830982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3590.850882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3594.956600] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3604.600074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3604.731892] Lustre: 119397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3604.771317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3604.775299] Lustre: Skipped 1 previous similar message [ 3604.826762] Lustre: lustre-MDT0001: new disk, initializing [ 3604.907911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3604.939413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3604.950320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3608.737474] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3616.090187] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3620.676164] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3626.586977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3626.865192] Lustre: lustre-OST0000: new disk, initializing [ 3626.869150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3626.878660] Lustre: 121374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3626.949784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.204622] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3634.215955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3634.233521] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3634.338513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3640.185539] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3644.296334] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3644.497298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3648.456688] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3648.643172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3654.625706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3654.635692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.651747] Lustre: Skipped 1 previous similar message [ 3654.654625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3656.386570] Lustre: server umount lustre-OST0000 complete [ 3660.261000] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.285454] Lustre: Skipped 2 previous similar messages [ 3668.011698] Lustre: server umount lustre-MDT0000 complete [ 3670.500858] LustreError: 119403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3670.530622] LustreError: 119403:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3671.329342] LustreError: 119388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686735 with bad export cookie 2536668690736978507 [ 3671.340375] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3671.353399] LustreError: 119388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3671.707304] Lustre: server umount lustre-MDT0001 complete [ 3678.305050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.696633] LustreError: 123587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3678.793945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3682.316684] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3683.810447] LustreError: 123588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.906070] LustreError: 123587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.881345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.844403] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3700.564376] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3706.661808] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3712.477209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3717.580454] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3722.693457] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3729.129877] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3733.477467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.479068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3733.494513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3733.500649] LustreError: Skipped 1 previous similar message [ 3733.512225] Lustre: Skipped 7 previous similar messages [ 3737.680419] Lustre: server umount lustre-OST0000 complete [ 3740.642385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3743.715315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.728876] Lustre: Skipped 2 previous similar messages [ 3746.733378] Lustre: server umount lustre-MDT0000 complete [ 3748.833104] LustreError: 123587:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3748.846835] LustreError: 123587:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3749.085501] LustreError: 123569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686813 with bad export cookie 2536668690736979389 [ 3749.087111] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3749.094168] LustreError: 123569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3749.278935] Lustre: server umount lustre-MDT0001 complete [ 3756.602357] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 3758.599531] Key type lgssc unregistered [ 3758.800992] LNet: 127216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3758.814416] LNetError: 127216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3758.824967] LNet: Removed LNI 192.168.203.109@tcp [ 3759.400372] Key type .llcrypt unregistered [ 3759.402142] Key type ._llcrypt unregistered [ 3771.114209] Key type ._llcrypt registered [ 3771.116202] Key type .llcrypt registered [ 3771.185258] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3781.689062] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3782.371288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3782.383212] alg: No test for adler32 (adler32-zlib) [ 3783.471702] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 3783.694669] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3785.384419] Key type lgssc registered [ 3786.301908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3811.335178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3811.363003] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3812.557353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3812.587721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3812.682852] Lustre: lustre-MDT0000: new disk, initializing [ 3812.755615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3812.775982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3816.178828] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3824.591407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3824.679272] Lustre: 130340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3824.782318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3824.790741] Lustre: Skipped 1 previous similar message [ 3824.864364] Lustre: lustre-MDT0001: new disk, initializing [ 3824.930282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3824.975062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3824.989294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3828.628466] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3835.716673] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3839.837978] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3845.109476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3845.327350] Lustre: lustre-OST0000: new disk, initializing [ 3845.330966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3845.337729] Lustre: 132309:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3845.427189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3849.796247] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3855.413284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3855.429362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3855.531568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3857.260804] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3860.811704] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3860.974912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3864.529517] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3864.691669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3871.205268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3871.205762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.219771] Lustre: Skipped 1 previous similar message [ 3871.221063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3871.234258] Lustre: Skipped 1 previous similar message [ 3872.443414] Lustre: server umount lustre-OST0000 complete [ 3876.326740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3876.334350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3876.337716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3876.349122] Lustre: Skipped 1 previous similar message [ 3881.541920] Lustre: server umount lustre-MDT0000 complete [ 3884.163625] LustreError: 130334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686948 with bad export cookie 4230668357691632778 [ 3884.168118] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3884.176496] LustreError: 130334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3884.440653] Lustre: server umount lustre-MDT0001 complete [ 3890.714959] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 19:15:54 (1789686954) [ 3895.247726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.575304] LustreError: 134719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3895.590775] LustreError: 134719:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3895.629173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3899.061977] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3900.897727] LustreError: 134720:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3904.996709] LustreError: 134719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.042991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3905.336445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3908.454424] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3914.214921] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3918.313769] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3920.869637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3920.878803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.892274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3920.902135] Lustre: Skipped 1 previous similar message [ 3924.962687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3924.962734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3930.083064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3930.086938] Lustre: Skipped 2 previous similar messages [ 3934.176300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3934.341862] Lustre: server umount lustre-MDT0000 complete [ 3936.227122] LustreError: 135446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3936.244520] LustreError: 135446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3937.062161] LustreError: 134700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687001 with bad export cookie 4230668357691633660 [ 3937.064572] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3937.073607] LustreError: 134700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3937.334315] Lustre: server umount lustre-MDT0001 complete [ 3943.239935] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 19:16:47 (1789687007) [ 3948.854228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3964.384874] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3969.504826] LustreError: 137133:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 3995.168306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3999.226908] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4004.532321] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4006.018991] Lustre: server umount lustre-OST0000 complete [ 4011.393295] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 19:17:55 (1789687075) [ 4016.002691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4016.322149] LustreError: 138578:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.373943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.294932] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4025.057189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.279646] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4033.345590] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4037.438285] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4041.835952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4046.461668] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4051.814379] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4066.793641] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.808035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4066.813946] Lustre: Skipped 3 previous similar messages [ 4071.394718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4072.807847] Lustre: server umount lustre-MDT0000 complete [ 4075.243713] LustreError: 139313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687139 with bad export cookie 4230668357691634325 [ 4075.244611] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4075.254177] LustreError: 139313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4075.435176] Lustre: server umount lustre-MDT0001 complete [ 4088.360912] Lustre: server umount lustre-OST0000 complete [ 4093.621556] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 19:19:17 (1789687157) [ 4099.561537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4100.093875] LustreError: 142256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4100.120066] LustreError: 142256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4100.174430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4100.187130] Lustre: Skipped 2 previous similar messages [ 4106.047701] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4116.450038] LustreError: 142256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.461461] LustreError: 142256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4116.915346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.729829] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4132.790571] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4142.878186] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4153.322912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4161.768688] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4170.010726] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4171.248800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4175.510402] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4175.667886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4180.168366] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4180.418083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4182.497653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4182.501917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.508648] Lustre: Skipped 2 previous similar messages [ 4182.512414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4182.515343] Lustre: Skipped 3 previous similar messages [ 4188.656784] Lustre: server umount lustre-OST0000 complete [ 4200.804178] Lustre: server umount lustre-MDT0000 complete [ 4201.958152] LustreError: 142257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.978835] LustreError: 142257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4204.846349] LustreError: 142237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687269 with bad export cookie 4230668357691635396 [ 4204.847688] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4204.857877] LustreError: 142237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4205.180712] Lustre: server umount lustre-MDT0001 complete [ 4213.354142] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 19:21:16 (1789687276) [ 4219.709827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4235.169645] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4240.288370] LustreError: 146418:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 4266.018512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4266.029516] Lustre: Skipped 2 previous similar messages [ 4271.964214] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4279.328932] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4285.418770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.897186] LustreError: 146443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4286.082961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4290.474032] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4298.154371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.837255] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4309.277894] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4314.014255] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4318.171570] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4318.337965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4322.240106] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4322.474670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4324.322893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4324.326026] LustreError: Skipped 1 previous similar message [ 4324.338861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4324.346181] Lustre: Skipped 3 previous similar messages [ 4324.352790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4324.358413] Lustre: Skipped 4 previous similar messages [ 4330.533433] Lustre: server umount lustre-MDT0000 complete [ 4333.858695] LustreError: 146424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687398 with bad export cookie 4230668357691636418 [ 4333.867693] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4333.884991] LustreError: 146424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4334.140254] Lustre: server umount lustre-MDT0001 complete [ 4347.931852] Lustre: server umount lustre-OST0000 complete [ 4354.875991] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 19:23:38 (1789687418) [ 4360.359771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4375.906384] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4381.024507] LustreError: 150610:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 4406.691865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4406.706854] Lustre: Skipped 2 previous similar messages [ 4412.533268] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4420.742637] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4427.312118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.471194] LustreError: 128564:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ab22a528 x1876622675428608/t0(0) o250->MGC192.168.203.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4427.789821] LustreError: 150635:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.810462] LustreError: 150635:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4427.922827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4433.150556] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4442.085720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4447.021895] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4454.988939] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4460.785349] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4467.281213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4467.526283] Lustre: lustre-OST0001: new disk, initializing [ 4467.531173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4467.552359] Lustre: 153479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4473.709821] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4474.451104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4474.464190] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4474.603995] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4483.088771] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4487.127965] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4487.290398] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4491.216594] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4491.461929] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4493.793734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4493.814844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4493.831800] Lustre: Skipped 2 previous similar messages [ 4493.844489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4493.862099] Lustre: Skipped 3 previous similar messages [ 4499.441617] Lustre: server umount lustre-OST0000 complete [ 4517.344189] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4517.618034] Lustre: server umount lustre-OST0001 complete [ 4527.892488] Lustre: server umount lustre-MDT0000 complete [ 4531.672142] LustreError: 150617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687595 with bad export cookie 4230668357691637370 [ 4531.680536] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4531.690853] LustreError: 150617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4532.107034] Lustre: server umount lustre-MDT0001 complete [ 4555.695773] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 19:26:58 (1789687618) [ 4557.628598] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4559.568390] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4561.772322] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 19:27:05 (1789687625) [ 4569.394719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.748456] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4569.775278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4574.284715] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4582.749498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4582.889654] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4582.946738] Lustre: 157043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4587.566462] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4594.832629] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4600.499678] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4608.329502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4608.672898] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4615.433244] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4618.364706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4622.842870] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4627.479526] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4627.662403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4631.917042] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4632.160281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4634.081112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4634.085855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4634.093996] LustreError: Skipped 4 previous similar messages [ 4634.104827] Lustre: Skipped 6 previous similar messages [ 4634.108276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4634.111105] Lustre: Skipped 12 previous similar messages [ 4640.328527] Lustre: server umount lustre-OST0000 complete [ 4662.464289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4662.827228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.850955] Lustre: Skipped 6 previous similar messages [ 4663.858353] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4663.874654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4663.877270] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4663.900682] Lustre: Skipped 1 previous similar message [ 4668.711677] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4676.281210] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4681.546641] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4681.786724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4686.496828] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4686.672446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4692.159842] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98cd8a2c6800.ost_server_uuid 50 [ 4693.616627] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff98cd8a2c6800.ost_server_uuid in FULL state after 0 sec [ 4693.776872] LustreError: 157049:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.9@tcp) returned error from blocking AST (req@0000000048a04f94 x1876622675535104 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000ac25c3d5/0x3ab65a74ceb207d4 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.9@tcp remote: 0xc5f74d555474f5de expref: 5 pid: 157050 timeout: 4793 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4693.824678] LustreError: lustre-MDT0000: A client on nid 192.168.203.9@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4693.836802] LustreError: 157039:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.9@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000ac25c3d5/0x3ab65a74ceb207d4 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.9@tcp remote: 0xc5f74d555474f5de expref: 6 pid: 157050 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4699.619913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4703.893283] Lustre: server umount lustre-OST0000 complete [ 4714.309976] Lustre: server umount lustre-MDT0000 complete [ 4714.977444] LustreError: 158496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4715.000317] LustreError: 158496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 4717.353366] LustreError: 157034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687781 with bad export cookie 4230668357691638553 [ 4717.356272] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4717.365582] LustreError: 157034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4723.808968] Lustre: server umount lustre-MDT0001 complete [ 4733.690527] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4736.386778] Key type lgssc unregistered [ 4736.622689] LNet: 163157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4736.627751] LNetError: 163157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4736.643643] LNet: Removed LNI 192.168.203.109@tcp [ 4737.455808] Key type .llcrypt unregistered [ 4737.461835] Key type ._llcrypt unregistered [ 4763.472248] Key type ._llcrypt registered [ 4763.477338] Key type .llcrypt registered [ 4763.632565] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4764.972693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4765.033503] alg: No test for adler32 (adler32-zlib) [ 4766.368168] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 4766.743065] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 4768.384222] Key type lgssc registered [ 4769.633787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4778.631565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4778.660590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.287923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4784.902144] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4790.762895] LustreError: 164568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4790.795410] LustreError: 164568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4794.982231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.409331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4801.565416] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4811.214256] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4817.448419] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4824.540489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4824.973778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4826.041274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4831.385071] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4837.933231] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4844.337926] Lustre: Failing over lustre-MDT0000 [ 4844.716261] Lustre: server umount lustre-MDT0000 complete [ 4845.547209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4845.557443] LustreError: 164565:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4845.567122] Lustre: Skipped 1 previous similar message [ 4845.614678] LustreError: 164565:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4850.659783] LustreError: 165296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4850.697904] LustreError: 165296:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4855.784630] LustreError: 164569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.805049] LustreError: 164569:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4856.941631] LustreError: 164548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.9@tcp arrived at 1789687921 with bad export cookie 10218648963221577906 [ 4856.962551] LustreError: 164548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4860.904119] LustreError: 164568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4860.925238] LustreError: 164568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4861.922569] Lustre: 163878:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789687909/real 1789687909] req@000000005ea33d17 x1876623706055936/t0(0) o400->MGC192.168.203.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789687925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4861.948452] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4863.987135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.162812] LustreError: 163875:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a98afdb8 x1876623706061184/t0(0) o250->MGC192.168.203.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4872.223894] LustreError: 165296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4872.251297] LustreError: 165296:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4872.476574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4872.530844] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4877.075192] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4877.801535] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4877.808228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4879.082530] Lustre: lustre-MDT0000: Denying connection for new client ff9ef88a-5514-49a5-8827-374ccca005c0 (at 192.168.203.9@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4888.033526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4888.040723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4888.041683] Lustre: Skipped 2 previous similar messages [ 4888.050812] Lustre: Skipped 1 previous similar message [ 4893.164170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4893.175851] Lustre: Skipped 1 previous similar message [ 4898.275312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4898.283032] Lustre: Skipped 1 previous similar message [ 4901.344137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4901.549743] Lustre: server umount lustre-OST0000 complete [ 4905.153039] LustreError: 168304:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4905.166657] Lustre: 167653:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4905.179410] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4905.193168] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4905.202303] LustreError: 167653:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000549fb4b1 x1876623706077184/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4905.239540] LustreError: 167653:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4905.257444] LustreError: 167653:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4905.277246] Lustre: lustre-MDT0000: Recovery over after 0:28, of 2 clients 0 recovered and 2 were evicted. [ 4905.661654] Lustre: server umount lustre-MDT0000 complete [ 4908.526100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4908.536259] LustreError: 167013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4909.227758] LustreError: 164548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687973 with bad export cookie 10218648963221578011 [ 4909.232139] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4909.245065] LustreError: 164548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4909.251038] LustreError: 163875:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4911.616822] Lustre: server umount lustre-MDT0001 complete [ 4923.241922] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4926.835546] Key type lgssc unregistered [ 4927.297228] LNet: 169058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4927.312199] LNetError: 169058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4927.323830] LNet: Removed LNI 192.168.203.109@tcp [ 4928.060126] Key type .llcrypt unregistered [ 4928.063878] Key type ._llcrypt unregistered [ 4952.943977] Key type ._llcrypt registered [ 4952.945572] Key type .llcrypt registered [ 4953.025652] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4953.686421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4953.714399] alg: No test for adler32 (adler32-zlib) [ 4954.707262] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 4954.922846] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 4956.568349] Key type lgssc registered [ 4957.536157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4964.464510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4964.486403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4965.922118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4970.550855] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4976.108547] LustreError: 170467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.129663] LustreError: 170467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4979.484929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.913294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4984.815266] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4992.544997] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4999.104423] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5005.694256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5006.170082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5011.343785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5012.609611] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5019.468332] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5026.789898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5026.794360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5026.811313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5030.883528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5030.886306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5030.899282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5030.923535] Lustre: Skipped 1 previous similar message [ 5032.327684] Lustre: server umount lustre-OST0000 complete [ 5036.006094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.022460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5041.127222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5041.135271] Lustre: Skipped 2 previous similar messages [ 5046.242786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.251361] Lustre: Skipped 1 previous similar message [ 5050.336747] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5050.587832] Lustre: server umount lustre-MDT0000 complete [ 5051.365828] LustreError: 170463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5051.378635] LustreError: 170463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5054.149611] LustreError: 170448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688118 with bad export cookie 13961937321757600413 [ 5054.157386] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5054.173941] LustreError: 170448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5054.647073] Lustre: server umount lustre-MDT0001 complete [ 5065.446526] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 5068.223787] Key type lgssc unregistered [ 5068.564619] LNet: 173999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5068.581939] LNetError: 173999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5069.613184] LNet: Removed LNI 192.168.203.109@tcp [ 5070.516925] Key type .llcrypt unregistered [ 5070.523106] Key type ._llcrypt unregistered [ 5100.390914] Key type ._llcrypt registered [ 5100.394513] Key type .llcrypt registered [ 5100.492829] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5101.453085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5101.607309] alg: No test for adler32 (adler32-zlib) [ 5102.701525] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 5103.027196] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 5104.688863] Key type lgssc registered [ 5106.067772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5113.346348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5113.387086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.847170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5119.313914] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5125.092906] LustreError: 175700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5125.115729] LustreError: 175700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5127.454631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5127.836709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5132.059654] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5139.090131] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5144.489336] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5150.566891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5150.974457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.623725] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5160.195904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5163.396263] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5170.659647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5170.662210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5170.682123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5170.714434] Lustre: Skipped 1 previous similar message [ 5175.782412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5175.790712] Lustre: Skipped 1 previous similar message [ 5176.432743] Lustre: server umount lustre-OST0000 complete [ 5180.899571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5180.910707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5180.913431] Lustre: Skipped 2 previous similar messages [ 5180.937941] Lustre: Skipped 1 previous similar message [ 5186.026552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5186.034809] Lustre: Skipped 1 previous similar message [ 5186.382950] Lustre: server umount lustre-MDT0000 complete [ 5189.955305] LustreError: 175681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688254 with bad export cookie 13043953599410235875 [ 5189.963973] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5189.972289] LustreError: 175681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5190.259557] Lustre: server umount lustre-MDT0001 complete [ 5203.503540] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 5206.385626] Key type lgssc unregistered [ 5206.779643] LNet: 179232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5206.793518] LNetError: 179232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5206.816732] LNet: Removed LNI 192.168.203.109@tcp [ 5207.654977] Key type .llcrypt unregistered [ 5207.659282] Key type ._llcrypt unregistered [ 5231.680939] Key type ._llcrypt registered [ 5231.688071] Key type .llcrypt registered [ 5231.821456] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5233.020067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5233.094207] alg: No test for adler32 (adler32-zlib) [ 5234.275989] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 5234.554688] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 5236.217194] Key type lgssc registered [ 5237.303958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5248.316735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5248.340352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5249.838250] Lustre: *** cfs_fail_loc=135, val=0*** [ 5249.944078] LustreError: 180750:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5249.954738] LustreError: 180750:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.109@tcp: cfg command failed: rc = -2 [ 5249.974538] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5249.974538] [ 5249.996578] LustreError: MGC192.168.203.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5250.021036] LustreError: 180710:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5250.032057] LustreError: 180710:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5250.052181] LustreError: 180710:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5250.058969] LustreError: 180710:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5250.142389] Lustre: server umount lustre-MDT0000 complete [ 5250.145834] LustreError: 180710:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5257.441140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.894919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5262.276649] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5268.452617] LustreError: 181120:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5268.480891] LustreError: 181120:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5270.678247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5271.098278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5275.935803] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5283.235864] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5288.080124] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5291.489069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5291.496460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5291.507845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5293.540487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.541119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5293.556982] Lustre: Skipped 1 previous similar message [ 5296.614972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5296.620393] Lustre: Skipped 1 previous similar message [ 5301.735269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5301.741389] Lustre: Skipped 1 previous similar message [ 5304.289171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5304.572474] Lustre: server umount lustre-MDT0000 complete [ 5306.850049] LustreError: 181115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.880839] LustreError: 181115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5308.145294] LustreError: 181103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688372 with bad export cookie 11364348192952665159 [ 5308.157491] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5308.419949] Lustre: server umount lustre-MDT0001 complete [ 5319.088908] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 5321.802120] Key type lgssc unregistered [ 5322.159398] LNet: 183441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5322.168084] LNetError: 183441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5322.196448] LNet: Removed LNI 192.168.203.109@tcp [ 5323.239188] Key type .llcrypt unregistered [ 5323.250746] Key type ._llcrypt unregistered [ 5345.420746] Key type ._llcrypt registered [ 5345.424636] Key type .llcrypt registered [ 5345.557617] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 5345.861311] Key type .llcrypt unregistered [ 5345.866770] Key type ._llcrypt unregistered [ 5362.201996] Key type ._llcrypt registered [ 5362.204831] Key type .llcrypt registered [ 5362.311545] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5363.294176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5363.394821] alg: No test for adler32 (adler32-zlib) [ 5364.590909] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 5364.940383] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 5366.672302] Key type lgssc registered [ 5367.648229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5375.191235] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5375.222172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5390.819792] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5395.939261] LustreError: 185564:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 5422.560216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5428.628454] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5436.218561] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5442.744543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5442.911894] LustreError: 184900:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000af2e62fd x1876624334129408/t0(0) o250->MGC192.168.203.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5443.208295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5443.300362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5447.791758] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5448.674405] LustreError: 185590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.695911] LustreError: 185590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5453.798720] LustreError: 185589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5455.425353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.766209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5459.696934] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5467.223393] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5473.228278] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5475.952442] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5489.123939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5489.136368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5489.150408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5491.682497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5491.689305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5491.715372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5492.728172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5492.742363] Lustre: Skipped 1 previous similar message [ 5494.334321] Lustre: server umount lustre-OST0000 complete [ 5501.922851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5501.945918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.976184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5504.403852] Lustre: server umount lustre-MDT0000 complete [ 5507.934779] LustreError: 185572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688572 with bad export cookie 3446296086984498088 [ 5507.942457] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5507.944989] LustreError: 185572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5508.299203] Lustre: server umount lustre-MDT0001 complete [ 5518.791638] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 5521.659340] Key type lgssc unregistered [ 5521.978070] LNet: 189579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5521.984698] LNetError: 189579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5521.995794] LNet: Removed LNI 192.168.203.109@tcp [ 5522.897777] Key type .llcrypt unregistered [ 5522.905308] Key type ._llcrypt unregistered [ 5543.786895] Key type ._llcrypt registered [ 5543.790409] Key type .llcrypt registered [ 5543.917116] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5544.931318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5544.973518] alg: No test for adler32 (adler32-zlib) [ 5546.154357] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 5546.460257] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 5548.136503] Key type lgssc registered [ 5549.285100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5556.256776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5556.282574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.693495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5562.484382] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5566.947765] LustreError: 190967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.971163] LustreError: 190967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5570.076204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5570.448349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5574.887743] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5582.409513] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5588.697171] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5595.622543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5596.199261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5601.671355] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5602.369991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5608.687405] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5613.204312] Lustre: Failing over lustre-MDT0000 [ 5613.456196] Lustre: server umount lustre-MDT0000 complete [ 5615.270463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.276188] LustreError: 190966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.294345] Lustre: Skipped 2 previous similar messages [ 5615.315768] LustreError: 190966:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5620.566330] LustreError: 190962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5620.581734] LustreError: 190962:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5625.692661] LustreError: 192208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5625.703967] LustreError: 192208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5630.807511] LustreError: 190962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5630.828206] LustreError: 190962:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5630.945387] Lustre: 190277:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789688679/real 1789688679] req@0000000088349c20 x1876624523944064/t0(0) o400->MGC192.168.203.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789688695 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5630.974736] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5631.543776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.053027] LustreError: 190963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.069812] LustreError: 190963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5641.195241] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5eb4a8e16874dda4 [ 5641.203345] Lustre: MGC192.168.203.109@tcp: Connection restored to 0@lo (at 0@lo) [ 5641.464848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5641.538639] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5645.886481] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5646.170694] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5646.838690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5646.865040] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5646.911256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5649.984385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5659.354361] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5673.959652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5673.970981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.984634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5674.988398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5675.000118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5675.009965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5675.836322] Lustre: server umount lustre-OST0000 complete [ 5682.659921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5682.660223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5682.664897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5682.694443] Lustre: Skipped 1 previous similar message [ 5684.920967] Lustre: server umount lustre-MDT0000 complete [ 5687.781885] LustreError: 190963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.805255] LustreError: 190963:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5688.118458] LustreError: 190947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688752 with bad export cookie 6824265021446741412 [ 5688.124941] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5688.139606] LustreError: 190947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5688.459990] Lustre: server umount lustre-MDT0001 complete [ 5698.733615] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 5700.995231] Key type lgssc unregistered [ 5701.277729] LNet: 195926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5701.292691] LNetError: 195926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5701.312817] LNet: Removed LNI 192.168.203.109@tcp [ 5701.953260] Key type .llcrypt unregistered [ 5701.958956] Key type ._llcrypt unregistered [ 5726.019381] Key type ._llcrypt registered [ 5726.021600] Key type .llcrypt registered [ 5726.161813] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5727.519590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5727.552570] alg: No test for adler32 (adler32-zlib) [ 5728.630125] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 5728.848699] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 5730.592263] Key type lgssc registered [ 5731.605376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5739.392087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5739.403148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.834387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5745.087384] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5751.272099] LustreError: 197335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.297933] LustreError: 197335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5753.245200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5753.568951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5757.790255] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5765.370958] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5770.464926] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5776.787879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5777.155170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5782.808946] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5786.367112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5789.516948] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5793.391198] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5801.195469] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5815.099876] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5819.365748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5819.377481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.404696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.433842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.435362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5825.452507] Lustre: server umount lustre-OST0000 complete [ 5830.626283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5830.635683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5830.660539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5830.671300] Lustre: Skipped 1 previous similar message [ 5832.673881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5832.677149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5835.005330] Lustre: server umount lustre-MDT0000 complete [ 5837.794489] LustreError: 198062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.819306] LustreError: 198062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5838.575471] LustreError: 197316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688902 with bad export cookie 14862232059637276764 [ 5838.577090] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5838.583338] LustreError: 197316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5838.876562] Lustre: server umount lustre-MDT0001 complete [ 5848.837124] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 5851.578169] Key type lgssc unregistered [ 5851.951644] LNet: 201184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5851.973390] LNetError: 201184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5851.999470] LNet: Removed LNI 192.168.203.109@tcp [ 5852.944988] Key type .llcrypt unregistered [ 5852.948277] Key type ._llcrypt unregistered [ 5874.036895] Key type ._llcrypt registered [ 5874.039493] Key type .llcrypt registered [ 5874.125369] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5875.002749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5875.027168] alg: No test for adler32 (adler32-zlib) [ 5876.118652] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 5876.334852] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 5878.001290] Key type lgssc registered [ 5879.105145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5886.762981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5886.790502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5888.215912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5892.720657] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5893.604110] LustreError: 202446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5893.622680] LustreError: 202446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5898.733531] LustreError: 202447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5901.697874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5902.068902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5906.645712] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5915.330647] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5921.352255] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5928.713566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5929.126657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5933.256733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5935.893989] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5943.746500] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5949.648552] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5963.264265] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5972.927936] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5979.613855] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5988.835126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5988.845802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.861136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5988.869663] Lustre: Skipped 1 previous similar message [ 5993.875674] Lustre: server umount lustre-OST0000 complete [ 5999.086845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5999.107617] Lustre: Skipped 1 previous similar message [ 5999.112845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5999.118837] Lustre: Skipped 1 previous similar message [ 6003.858139] Lustre: server umount lustre-MDT0000 complete [ 6004.193962] LustreError: 202447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6004.217058] LustreError: 202447:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6007.319543] LustreError: 202425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689071 with bad export cookie 17136244479132499952 [ 6007.320460] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6007.333151] LustreError: 202425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6007.865745] Lustre: server umount lustre-MDT0001 complete [ 6019.576510] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 6022.394770] Key type lgssc unregistered [ 6022.731678] LNet: 207469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6022.737720] LNetError: 207469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6022.765747] LNet: Removed LNI 192.168.203.109@tcp [ 6023.695739] Key type .llcrypt unregistered [ 6023.698246] Key type ._llcrypt unregistered [ 6047.460409] Key type ._llcrypt registered [ 6047.466426] Key type .llcrypt registered [ 6047.577417] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 6048.584964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6048.682381] alg: No test for adler32 (adler32-zlib) [ 6049.744245] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 6049.965676] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 6051.648358] Key type lgssc registered [ 6052.600697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6059.637252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6059.654049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.131155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6065.467940] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6071.275481] LustreError: 208879:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6071.302913] LustreError: 208879:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6074.279745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6074.720969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6079.676719] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6086.954463] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6092.980065] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6101.811579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6102.476604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6106.643169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6109.838818] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6117.732772] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6134.243235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6134.275660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6134.320034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6137.319797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6137.331485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6137.344469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6140.035820] Lustre: server umount lustre-OST0000 complete [ 6146.529420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6146.543074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.570312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6146.573289] Lustre: Skipped 2 previous similar messages [ 6148.585298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.594118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6148.646380] Lustre: Skipped 1 previous similar message [ 6151.518649] Lustre: server umount lustre-MDT0000 complete [ 6153.706421] LustreError: 208878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.718621] LustreError: 208878:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6156.129343] LustreError: 208858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689220 with bad export cookie 7917860914219167308 [ 6156.140169] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6156.156569] LustreError: 208858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6156.430733] Lustre: server umount lustre-MDT0001 complete [ 6170.090954] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 6173.241986] Key type lgssc unregistered [ 6173.502622] LNet: 212561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6173.521632] LNetError: 212561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6173.546626] LNet: Removed LNI 192.168.203.109@tcp [ 6174.498993] Key type .llcrypt unregistered [ 6174.501447] Key type ._llcrypt unregistered [ 6196.876924] Key type ._llcrypt registered [ 6196.878503] Key type .llcrypt registered [ 6197.022370] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 6198.743284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6198.860657] alg: No test for adler32 (adler32-zlib) [ 6200.546356] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 6201.294524] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 6203.177278] Key type lgssc registered [ 6205.494453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6215.377908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6215.407614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6216.864329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6220.639307] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6227.429476] LustreError: 213822:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6227.466082] LustreError: 213822:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6230.154881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6230.498583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6234.782850] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6242.432714] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6249.014107] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6256.280598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6256.644477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6262.785637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6263.585396] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6270.806822] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6285.283610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6285.311811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6285.312614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6285.332013] Lustre: Skipped 1 previous similar message [ 6290.402291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6290.414743] Lustre: Skipped 1 previous similar message [ 6290.867848] Lustre: server umount lustre-OST0000 complete [ 6295.524701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.562467] Lustre: Skipped 1 previous similar message [ 6295.568243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6295.578643] Lustre: Skipped 1 previous similar message [ 6300.641580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6300.661331] Lustre: Skipped 1 previous similar message [ 6305.774576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6305.792583] Lustre: Skipped 1 previous similar message [ 6309.344392] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6309.682306] Lustre: server umount lustre-MDT0000 complete [ 6310.883027] LustreError: 215668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.903484] LustreError: 215668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6313.557244] LustreError: 213805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689377 with bad export cookie 13078869166042707244 [ 6313.566238] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6313.570937] LustreError: 213805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6314.033941] Lustre: server umount lustre-MDT0001 complete [ 6328.668532] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 6332.495778] Key type lgssc unregistered [ 6332.962929] LNet: 217507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6332.974273] LNetError: 217507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6333.005474] LNet: Removed LNI 192.168.203.109@tcp [ 6334.259790] Key type .llcrypt unregistered [ 6334.261898] Key type ._llcrypt unregistered [ 6356.440563] Key type ._llcrypt registered [ 6356.443551] Key type .llcrypt registered [ 6356.580489] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 6357.769429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6357.826679] alg: No test for adler32 (adler32-zlib) [ 6358.897987] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 6359.168226] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 6360.872366] Key type lgssc registered [ 6362.101992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6370.055449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6370.090744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.750779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6377.021709] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6381.027830] LustreError: 218771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.044662] LustreError: 218771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6386.146030] LustreError: 218772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.324429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6386.771110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6392.256384] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6401.204173] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6407.820901] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6416.643111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6417.130315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6423.883997] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6432.326620] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6441.720714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6453.480581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.597415] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6453.917324] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6462.185515] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6470.401821] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6472.313807] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6484.339887] Lustre: Permanently deactivating lustre-OST0001 [ 6484.381903] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6491.230974] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6501.123454] Lustre: Permanently reactivating lustre-OST0001 [ 6501.141346] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6501.146686] Lustre: Skipped 2 previous similar messages [ 6509.144820] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.162340] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6509.170808] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6509.194033] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6509.246213] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6511.912602] Lustre: server umount lustre-OST0001 complete [ 6513.651906] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.674813] Lustre: Skipped 1 previous similar message [ 6513.680742] LustreError: 221270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6513.709374] LustreError: 221270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6518.753491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6518.757635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6518.758391] LustreError: 220594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6518.768953] LustreError: Skipped 1 previous similar message [ 6518.779319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6518.784103] Lustre: Skipped 2 previous similar messages [ 6518.819242] Lustre: Skipped 1 previous similar message [ 6522.292730] Lustre: server umount lustre-OST0000 complete [ 6528.996361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6529.018133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6529.024496] Lustre: Skipped 1 previous similar message [ 6533.328761] Lustre: server umount lustre-MDT0000 complete [ 6534.113187] LustreError: 218772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6534.132247] LustreError: 218772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6537.432184] LustreError: 218752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689601 with bad export cookie 16524930237331433407 [ 6537.438209] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6537.443169] LustreError: 218752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6538.143086] Lustre: server umount lustre-MDT0001 complete [ 6550.275308] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 6553.400233] Key type lgssc unregistered [ 6553.734631] LNet: 223863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6553.740625] LNetError: 223863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6553.767182] LNet: Removed LNI 192.168.203.109@tcp [ 6554.738351] Key type .llcrypt unregistered [ 6554.739718] Key type ._llcrypt unregistered [ 6597.895290] Key type ._llcrypt registered [ 6597.897388] Key type .llcrypt registered [ 6597.992661] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 6599.428210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6599.479354] alg: No test for adler32 (adler32-zlib) [ 6600.579155] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 6600.940536] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 6602.656223] Key type lgssc registered [ 6604.088696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6612.072633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6612.103888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6613.379279] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6613.395076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6613.492429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6617.658907] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6626.574579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6626.723748] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6626.767735] Lustre: 226079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6627.057390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6631.893323] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6641.531516] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6649.775347] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6658.788688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6659.053196] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6659.417497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6665.759566] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6668.426304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6674.434753] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6678.358193] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6683.161135] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6692.522336] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6700.381616] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6711.599572] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6711.605313] Lustre: Skipped 1 previous similar message [ 6737.281218] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6737.286695] Lustre: Skipped 2 previous similar messages [ 6771.542539] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6771.548435] Lustre: Skipped 4 previous similar messages [ 6842.341834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6842.353325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.367879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6842.855027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6842.863413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.883693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6846.788231] Lustre: server umount lustre-OST0000 complete [ 6852.586184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6852.604092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.617059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6856.791064] Lustre: server umount lustre-MDT0000 complete [ 6861.083101] LustreError: 226072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689925 with bad export cookie 18159101932678765812 [ 6861.086721] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6861.097408] LustreError: 226072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6861.483975] Lustre: server umount lustre-MDT0001 complete [ 6873.806782] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 6876.871676] Key type lgssc unregistered [ 6877.293939] LNet: 230889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6877.303457] LNetError: 230889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6877.320538] LNet: Removed LNI 192.168.203.109@tcp [ 6878.119611] Key type .llcrypt unregistered [ 6878.122523] Key type ._llcrypt unregistered [ 6901.407632] Key type ._llcrypt registered [ 6901.411224] Key type .llcrypt registered [ 6901.503778] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 6902.342153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6902.352860] alg: No test for adler32 (adler32-zlib) [ 6903.502838] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 6903.816074] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 6905.536297] Key type lgssc registered [ 6906.606815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6914.070191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6914.101988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.472814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6919.678478] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6925.796347] LustreError: 232301:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6925.807543] LustreError: 232301:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6927.978143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.544615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6934.228243] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6943.634756] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6950.501608] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6957.423885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6957.822078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6960.889557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6963.462074] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6970.433591] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6986.360893] Lustre: server umount lustre-OST0000 complete [ 6993.890875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6993.910905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6993.924626] Lustre: Skipped 1 previous similar message [ 6993.965873] Lustre: Skipped 1 previous similar message [ 6999.011799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6999.026891] Lustre: Skipped 1 previous similar message [ 7000.245362] Lustre: server umount lustre-MDT0000 complete [ 7004.064153] Lustre: 231609:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789690052/real 1789690052] req@00000000fdb4649c x1876625947922432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789690068 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7004.084397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7004.094086] LustreError: 232301:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7004.108791] LustreError: 232301:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7004.416818] LustreError: 232281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789690068 with bad export cookie 6371733352182612598 [ 7004.419955] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7004.434026] LustreError: 232281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7005.078690] Lustre: server umount lustre-MDT0001 complete [ 7019.456720] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 7022.667554] Key type lgssc unregistered [ 7023.101718] LNet: 235989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7023.112516] LNetError: 235989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7023.139249] LNet: Removed LNI 192.168.203.109@tcp [ 7024.314264] Key type .llcrypt unregistered [ 7024.317819] Key type ._llcrypt unregistered [ 7047.058440] Key type ._llcrypt registered [ 7047.066806] Key type .llcrypt registered [ 7047.197974] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7048.648783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7048.703099] alg: No test for adler32 (adler32-zlib) [ 7049.894453] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 7050.245815] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 7051.968389] Key type lgssc registered [ 7053.500650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7323.149812] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 7325.970663] Key type lgssc unregistered [ 7326.365838] LNet: 238006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7326.376788] LNetError: 238006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7326.401915] LNet: Removed LNI 192.168.203.109@tcp [ 7327.393753] Key type .llcrypt unregistered [ 7327.396297] Key type ._llcrypt unregistered [ 7354.483052] Key type ._llcrypt registered [ 7354.484882] Key type .llcrypt registered [ 7354.586229] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 7372.868484] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7374.485503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7374.524310] alg: No test for adler32 (adler32-zlib) [ 7375.660948] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 7376.007464] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 7377.760319] Key type lgssc registered [ 7378.950513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7416.102894] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7141 sec ========== 20:14:38 (1789690478) [ 7418.905214] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:14:41 (1789690481) === [ 7423.196603] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:14:46 (1789690486) === [ 7465.954954] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 7468.849649] Key type lgssc unregistered [ 7469.152214] LNet: 243238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7469.158313] LNetError: 243238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7469.177234] LNet: Removed LNI 192.168.203.109@tcp [ 7469.981652] Key type .llcrypt unregistered [ 7469.984342] Key type ._llcrypt unregistered