[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 469888788 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003314] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.010156] LSM: Security Framework initializing [ 0.011119] Yama: becoming mindful. [ 0.013002] SELinux: Initializing. [ 0.014090] *** VALIDATE selinux *** [ 0.023162] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027749] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] *** VALIDATE tmpfs *** [ 0.031540] *** VALIDATE proc *** [ 0.032211] *** VALIDATE cgroup *** [ 0.034002] *** VALIDATE cgroup2 *** [ 0.034943] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035229] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042410] debug: unmapping init [mem 0xffffffff8ba59000-0xffffffff8ba60fff] [ 0.045204] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046817] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050017] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052019] ... fixed-purpose events: 3 [ 0.053017] ... event mask: 000000070000000f [ 0.054308] rcu: Hierarchical SRCU implementation. [ 0.056519] smp: Bringing up secondary CPUs ... [ 0.057482] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.062810] smp: Brought up 1 node, 4 CPUs [ 0.064027] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130660] node 0 deferred pages initialised in 65ms [ 0.133173] devtmpfs: initialized [ 0.134265] x86/mm: Memory block size: 128MB [ 0.136645] gcov: version magic: 0x41383552 [ 0.139150] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143327] pinctrl core: initialized pinctrl subsystem [ 0.144142] [ 0.144473] ************************************************************* [ 0.146009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147008] ** ** [ 0.149011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151011] ** ** [ 0.152010] ** This means that this kernel is built to expose internal ** [ 0.154009] ** IOMMU data structures, which may compromise security on ** [ 0.156008] ** your system. ** [ 0.157012] ** ** [ 0.159009] ** If you see this message and you are not debugging the ** [ 0.161014] ** kernel, report this immediately to your vendor! ** [ 0.163011] ** ** [ 0.165009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166009] ************************************************************* [ 0.168647] NET: Registered protocol family 16 [ 0.170344] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177135] cpuidle: using governor menu [ 0.178409] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181697] PCI: Using configuration type 1 for base access [ 0.184259] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199151] cryptd: max_cpu_qlen set to 1000 [ 0.202179] ACPI: Added _OSI(Module Device) [ 0.203011] ACPI: Added _OSI(Processor Device) [ 0.204015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209174] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.214215] ACPI: Interpreter enabled [ 0.215041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215938] ACPI: Using IOAPIC for interrupt routing [ 0.216041] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219324] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226795] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.228029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.229012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236038] acpiphp: Slot [2] registered [ 0.236997] acpiphp: Slot [5] registered [ 0.237997] acpiphp: Slot [6] registered [ 0.239007] acpiphp: Slot [7] registered [ 0.239912] acpiphp: Slot [8] registered [ 0.241067] acpiphp: Slot [9] registered [ 0.242030] acpiphp: Slot [10] registered [ 0.243058] acpiphp: Slot [3] registered [ 0.244000] acpiphp: Slot [4] registered [ 0.245095] acpiphp: Slot [11] registered [ 0.246114] acpiphp: Slot [12] registered [ 0.247126] acpiphp: Slot [13] registered [ 0.249103] acpiphp: Slot [14] registered [ 0.250166] acpiphp: Slot [15] registered [ 0.252129] acpiphp: Slot [16] registered [ 0.253110] acpiphp: Slot [17] registered [ 0.255116] acpiphp: Slot [18] registered [ 0.256099] acpiphp: Slot [19] registered [ 0.258173] acpiphp: Slot [20] registered [ 0.259104] acpiphp: Slot [21] registered [ 0.261125] acpiphp: Slot [22] registered [ 0.263142] acpiphp: Slot [23] registered [ 0.264169] acpiphp: Slot [24] registered [ 0.266071] acpiphp: Slot [25] registered [ 0.267007] acpiphp: Slot [26] registered [ 0.268118] acpiphp: Slot [27] registered [ 0.269095] acpiphp: Slot [28] registered [ 0.270057] acpiphp: Slot [29] registered [ 0.271057] acpiphp: Slot [30] registered [ 0.272156] acpiphp: Slot [31] registered [ 0.274112] PCI host bridge to bus 0000:00 [ 0.275007] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277084] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.279023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.282033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.285026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.288031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.290195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.293172] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.297540] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.305015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.308055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.311017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.312012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.314012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316410] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.317688] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.321048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.322683] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.327022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.336083] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.339013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.345385] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.355015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.363018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.382018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.395104] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.400016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.403016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.416027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.425981] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.431012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.435018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.445017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.454693] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.465013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.469013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.481015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.492194] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.497014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.504015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.514014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.521281] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.525014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.531017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.540013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.547495] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.550262] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.552430] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.554292] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.556234] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.561102] iommu: Default domain type: Passthrough [ 0.563434] SCSI subsystem initialized [ 0.564221] ACPI: bus type USB registered [ 0.566087] usbcore: registered new interface driver usbfs [ 0.567072] usbcore: registered new interface driver hub [ 0.568073] usbcore: registered new device driver usb [ 0.570179] pps_core: LinuxPPS API ver. 1 registered [ 0.571009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.574079] PTP clock support registered [ 0.575127] EDAC MC: Ver: 3.0.0 [ 0.577071] PCI: Using ACPI for IRQ routing [ 0.578951] NetLabel: Initializing [ 0.579052] NetLabel: domain hash size = 128 [ 0.581008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.582060] NetLabel: unlabeled traffic allowed by default [ 0.584037] vgaarb: loaded [ 0.584750] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.586009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.591469] clocksource: Switched to clocksource kvm-clock [ 0.683412] VFS: Disk quotas dquot_6.6.0 [ 0.684867] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.687081] *** VALIDATE ramfs *** [ 0.688121] *** VALIDATE hugetlbfs *** [ 0.690062] pnp: PnP ACPI init [ 0.691959] pnp: PnP ACPI: found 6 devices [ 0.707998] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.711148] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.713040] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.714743] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.716886] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.719121] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.722103] NET: Registered protocol family 2 [ 0.724555] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.729089] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.732290] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.735830] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.738846] TCP: Hash tables configured (established 65536 bind 65536) [ 0.741113] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.743927] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.746054] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.747868] NET: Registered protocol family 1 [ 0.749727] RPC: Registered named UNIX socket transport module. [ 0.751031] RPC: Registered udp transport module. [ 0.752358] RPC: Registered tcp transport module. [ 0.753654] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.755511] NET: Registered protocol family 44 [ 0.756969] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.758699] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.759849] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.761119] PCI: CLS 0 bytes, default 64 [ 0.762136] Unpacking initramfs... [ 2.099876] debug: unmapping init [mem 0xffff9abcfcc54000-0xffff9abcfffbffff] [ 2.103784] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.106018] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.108906] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.576534] Initialise system trusted keyrings [ 2.577780] Key type blacklist registered [ 2.579625] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.586632] zbud: loaded [ 2.588881] *** VALIDATE nfs *** [ 2.589797] *** VALIDATE nfs4 *** [ 2.591171] pstore: using deflate compression [ 2.594260] Platform Keyring initialized [ 2.690647] NET: Registered protocol family 38 [ 2.692861] Key type asymmetric registered [ 2.694523] Asymmetric key parser 'x509' registered [ 2.696571] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.699937] io scheduler mq-deadline registered [ 2.701831] io scheduler kyber registered [ 2.704260] io scheduler bfq registered [ 2.705876] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.709271] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.711930] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.714573] ACPI: Power Button [PWRF] [ 2.720078] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.727516] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.738074] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.744189] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.762528] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.789407] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.818214] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.822597] Non-volatile memory driver v1.3 [ 2.824249] Linux agpgart interface v0.103 [ 2.855126] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.857477] vda: detected capacity change from 0 to 76677120 [ 2.870755] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.873750] vdb: detected capacity change from 0 to 1073741824 [ 2.885130] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.886920] vdc: detected capacity change from 0 to 2621440000 [ 2.898660] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.901387] vdd: detected capacity change from 0 to 2621440000 [ 2.917597] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.920569] vde: detected capacity change from 0 to 4294967296 [ 2.938424] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.941636] vdf: detected capacity change from 0 to 4294967296 [ 2.947824] libphy: Fixed MDIO Bus: probed [ 2.954390] usbcore: registered new interface driver usbserial_generic [ 2.957244] usbserial: USB Serial support registered for generic [ 2.960319] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.963651] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.964748] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.966536] mousedev: PS/2 mouse device common for all mice [ 2.968458] rtc_cmos 00:05: RTC can wake from S4 [ 2.970259] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.973470] rtc_cmos 00:05: registered as rtc0 [ 2.974406] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.975775] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.978621] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.980066] intel_pstate: CPU model not supported [ 2.985116] hid: raw HID events driver (C) Jiri Kosina [ 2.987040] usbcore: registered new interface driver usbhid [ 2.989123] usbhid: USB HID core driver [ 2.990790] drop_monitor: Initializing network drop monitor service [ 2.993156] Initializing XFRM netlink socket [ 2.995270] NET: Registered protocol family 10 [ 2.998210] Segment Routing with IPv6 [ 2.999863] NET: Registered protocol family 17 [ 3.001900] mpls_gso: MPLS GSO support [ 3.007448] RAS: Correctable Errors collector initialized. [ 3.009718] AVX version of gcm_enc/dec engaged. [ 3.011506] AES CTR mode by8 optimization enabled [ 3.089536] sched_clock: Marking stable (3089512136, 0)->(3922181193, -832669057) [ 3.093174] registered taskstats version 1 [ 3.095201] Loading compiled-in X.509 certificates [ 3.097261] zswap: loaded using pool lzo/zbud [ 3.122388] Key type big_key registered [ 3.133036] Key type encrypted registered [ 3.134797] ima: No TPM chip found, activating TPM-bypass! [ 3.136982] ima: Allocated hash algorithm: sha1 [ 3.138853] ima: No architecture policies found [ 3.140753] evm: Initialising EVM extended attributes: [ 3.142797] evm: security.selinux [ 3.144157] evm: security.ima [ 3.145423] evm: security.capability [ 3.146847] evm: HMAC attrs: 0x1 [ 3.149308] rtc_cmos 00:05: setting system clock to 2026-09-03 12:10:34 UTC (1788437434) [ 3.156446] debug: unmapping init [mem 0xffffffff8ca03000-0xffffffff8cbfffff] [ 3.159764] debug: unmapping init [mem 0xffffffff8b782000-0xffffffff8ba58fff] [ 3.167472] Write protecting the kernel read-only data: 28672k [ 3.171301] debug: unmapping init [mem 0xffffffff89e03000-0xffffffff89ffffff] [ 3.173989] debug: unmapping init [mem 0xffffffff8a714000-0xffffffff8a7fffff] [ 3.208981] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.217386] systemd[1]: Detected virtualization kvm. [ 3.218928] systemd[1]: Detected architecture x86-64. [ 3.220257] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.245799] systemd[1]: No hostname configured. [ 3.247363] systemd[1]: Set hostname to . [ 3.249276] random: systemd: uninitialized urandom read (16 bytes read) [ 3.250890] systemd[1]: Initializing machine ID from random generator. [ 3.295207] random: ln: uninitialized urandom read (6 bytes read) [ 3.359668] random: systemd: uninitialized urandom read (16 bytes read) [ 3.362816] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.368538] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.374059] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.856520] device-mapper: uevent: version 1.0.3 [ 3.859094] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Ker[ 4.122053] random: fast init done nel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 4.503134] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.562877] scsi host0: ata_piix [ 4.581234] scsi host1: ata_piix [ 4.583276] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.586653] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.308437] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.654800] random: crng init done [ 9.656705] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.962775] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.134943] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.370841] SELinux: Disabled at runtime. [ 11.430580] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.439226] systemd[1]: Detected virtualization kvm. [ 11.440769] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.912758] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.916466] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.920902] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.924496] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.927935] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.938227] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.964327] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ 12.138701] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.419534] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.679846] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.712424] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.826510] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.836261] EDAC sbridge: Ver: 1.1.2 [ 14.277168] Key type dns_resolver registered [ 14.580462] NFS: Registering the id_resolver key type [ 14.582629] Key type id_resolver registered [ 14.584357] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg225-server login: [ 39.151585] libcfs: loading out-of-tree module taints kernel. [ 39.168543] Key type ._llcrypt registered [ 39.169580] Key type .llcrypt registered [ 39.210378] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 47.995229] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 48.579172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.584835] alg: No test for adler32 (adler32-zlib) [ 49.595350] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 49.935487] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 51.551188] Key type lgssc registered [ 53.563158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.277188] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 104.113072] hrtimer: interrupt took 7061038 ns [ 123.089358] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 139.194972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 139.227986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.514044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 140.567822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 140.640447] Lustre: lustre-MDT0000: new disk, initializing [ 140.731712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.744693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.983932] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.576849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.740352] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 161.791934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 161.800450] Lustre: Skipped 1 previous similar message [ 161.892611] Lustre: lustre-MDT0001: new disk, initializing [ 161.965854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.004361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 162.013899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 166.678224] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.981129] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 182.290573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 182.666849] Lustre: lustre-OST0000: new disk, initializing [ 182.681694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 182.695585] Lustre: 8423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 182.781146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.110222] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.550135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 190.568369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 190.711487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.737458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.039125] Lustre: lustre-OST0001: new disk, initializing [ 206.053754] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 206.070758] Lustre: 9493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.167732] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.541405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.560945] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.651107] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 214.138313] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.374648] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 237.867314] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 245.332881] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.441566] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing yml_node [ 256.167245] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 259.185845] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 262.368988] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 264.400614] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:14:53 EDT 2026 [ 286.804934] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 288.899904] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 305.632280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.633031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 305.637502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.657899] Lustre: Skipped 2 previous similar messages [ 310.754468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.770040] Lustre: Skipped 6 previous similar messages [ 311.674404] Lustre: server umount lustre-MDT0000 complete [ 315.541821] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437746 with bad export cookie 18160908187002211170 [ 315.542993] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.552758] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 315.883318] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.891509] Lustre: Skipped 2 previous similar messages [ 315.898792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 315.904962] Lustre: Skipped 1 previous similar message [ 320.999899] LustreError: 6492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.000288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.024278] LustreError: 6492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 321.056691] Lustre: Skipped 1 previous similar message [ 322.103163] Lustre: server umount lustre-MDT0001 complete [ 326.816423] Lustre: server umount lustre-OST0000 complete [ 330.568645] Lustre: server umount lustre-OST0001 complete [ 337.543288] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 350.493789] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 391.172834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.459776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 391.506842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 391.587486] Lustre: lustre-MDT0000: new disk, initializing [ 391.659631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 391.676401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 397.017271] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 411.641906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.853219] Lustre: 15049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 411.950782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 411.961632] Lustre: Skipped 1 previous similar message [ 412.136887] Lustre: lustre-MDT0001: new disk, initializing [ 412.297165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.376529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.387700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.897301] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 430.417794] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 437.418092] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 444.157330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 444.509630] Lustre: lustre-OST0000: new disk, initializing [ 444.519689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 444.541293] Lustre: 17017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 444.693351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.572341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 446.591499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 446.713876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 452.646834] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 464.508723] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 469.498870] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 469.756973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 474.547509] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 474.977326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 477.670671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.672654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.707116] Lustre: Skipped 1 previous similar message [ 477.731352] Lustre: Skipped 1 previous similar message [ 482.745552] Lustre: server umount lustre-OST0000 complete [ 487.419421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.459858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 487.479932] Lustre: Skipped 1 previous similar message [ 494.201968] Lustre: server umount lustre-MDT0000 complete [ 497.636516] LustreError: 15057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.663414] LustreError: 15057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 498.660047] LustreError: 15062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.674843] LustreError: 15062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 499.217864] LustreError: 17027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437930 with bad export cookie 18160908187002212703 [ 499.221695] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.236175] LustreError: 17027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 499.760645] Lustre: server umount lustre-MDT0001 complete [ 505.721782] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 08:18:53 (1788437933) [ 514.710149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.325171] LustreError: 19287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.397480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.172735] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 520.678384] LustreError: 19288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.310205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.486103] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 544.119669] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 550.979765] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 558.497315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 565.901443] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 575.027245] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 584.683330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.698950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 584.710688] Lustre: Skipped 2 previous similar messages [ 584.733727] Lustre: Skipped 3 previous similar messages [ 589.405894] Lustre: server umount lustre-OST0000 complete [ 594.912437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.932660] Lustre: Skipped 1 previous similar message [ 608.223111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 608.382946] Lustre: server umount lustre-MDT0000 complete [ 610.274986] LustreError: 20016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.296695] LustreError: 20016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 612.147278] LustreError: 19267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438043 with bad export cookie 18160908187002213578 [ 612.150463] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.167450] LustreError: 19267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 612.606729] Lustre: server umount lustre-MDT0001 complete [ 621.783785] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 624.089170] Key type lgssc unregistered [ 624.299811] LNet: 22822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.304477] LNetError: 22822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.337488] LNet: Removed LNI 192.168.202.125@tcp [ 625.212475] Key type .llcrypt unregistered [ 625.215885] Key type ._llcrypt unregistered [ 655.995759] Key type ._llcrypt registered [ 655.997707] Key type .llcrypt registered [ 656.115608] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 657.374555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 657.459595] alg: No test for adler32 (adler32-zlib) [ 658.927938] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 659.266065] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 661.119936] Key type lgssc registered [ 662.848630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 672.286228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 672.308364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.862449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 678.907185] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 684.003427] LustreError: 24240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 684.015176] LustreError: 24240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 689.137091] LustreError: 24239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.025539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.429913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 695.879164] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 705.058317] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 711.304715] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 719.761661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 720.181978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 727.132738] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 734.600690] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 743.771165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 754.148690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.149069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 754.166500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 754.183086] Lustre: Skipped 1 previous similar message [ 758.732289] Lustre: server umount lustre-OST0000 complete [ 764.394792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 764.404769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 764.412265] Lustre: Skipped 2 previous similar messages [ 764.422059] Lustre: Skipped 1 previous similar message [ 769.504505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 769.513604] Lustre: Skipped 1 previous similar message [ 774.626548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.631797] Lustre: Skipped 1 previous similar message [ 777.186074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 777.566456] Lustre: server umount lustre-MDT0000 complete [ 779.746553] LustreError: 24239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 779.776191] LustreError: 24239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 781.652450] LustreError: 24221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438213 with bad export cookie 14945537725693651708 [ 781.685115] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 781.690685] LustreError: 24221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 782.287202] Lustre: server umount lustre-MDT0001 complete [ 795.938713] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 799.921212] Key type lgssc unregistered [ 800.258726] LNet: 28218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.265361] LNetError: 28218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.277492] LNet: Removed LNI 192.168.202.125@tcp [ 801.377322] Key type .llcrypt unregistered [ 801.381109] Key type ._llcrypt unregistered [ 830.602745] Key type ._llcrypt registered [ 830.613124] Key type .llcrypt registered [ 830.867957] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 833.179578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 833.695136] alg: No test for adler32 (adler32-zlib) [ 835.273652] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 835.635159] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 837.303343] Key type lgssc registered [ 838.715487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 848.458045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 848.490493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 850.045767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 856.013682] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 860.645880] LustreError: 29628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.660503] LustreError: 29628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 865.763268] LustreError: 29629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 866.242862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.724022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 872.643171] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 880.499745] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 887.032657] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 902.528228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 903.082307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.083922] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 921.465768] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 921.608689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 931.810281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.814233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.817483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 931.836041] Lustre: Skipped 1 previous similar message [ 934.211535] Lustre: server umount lustre-OST0000 complete [ 938.471616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 938.487415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.509628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 938.518050] Lustre: Skipped 1 previous similar message [ 942.050706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 942.050959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.178110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 947.184248] Lustre: Skipped 2 previous similar messages [ 952.287629] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 952.295649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 952.328466] Lustre: Skipped 1 previous similar message [ 952.582742] Lustre: server umount lustre-MDT0000 complete [ 957.412667] LustreError: 29623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.436539] LustreError: 29623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 957.468237] LustreError: 29611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438388 with bad export cookie 3748328736407296318 [ 957.473362] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 957.477969] LustreError: 29611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 958.119502] Lustre: server umount lustre-MDT0001 complete [ 971.529352] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 974.758848] Key type lgssc unregistered [ 975.048836] LNet: 33511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.063086] LNetError: 33511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.133749] LNet: Removed LNI 192.168.202.125@tcp [ 975.986684] Key type .llcrypt unregistered [ 975.993849] Key type ._llcrypt unregistered [ 1008.218743] Key type ._llcrypt registered [ 1008.221105] Key type .llcrypt registered [ 1008.409581] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 1009.738283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1009.827041] alg: No test for adler32 (adler32-zlib) [ 1011.448680] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1011.881847] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 1013.597378] Key type lgssc registered [ 1015.185588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1023.778133] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1023.790187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.300575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.753541] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1034.731905] LustreError: 34924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.749906] LustreError: 34924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1039.840236] LustreError: 34923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.950007] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.221997] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1044.976852] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1053.725748] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1060.742536] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1068.784386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1069.489935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.531452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1077.427537] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1086.333053] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1096.161405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1096.163892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.172177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.265652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1100.268082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.303425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1102.272558] Lustre: server umount lustre-OST0000 complete [ 1106.926932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1106.938092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.955306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.964875] Lustre: Skipped 1 previous similar message [ 1110.496594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1110.499527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.145319] Lustre: server umount lustre-MDT0000 complete [ 1115.628368] LustreError: 34924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.657613] LustreError: 34924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1118.945544] LustreError: 35657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438550 with bad export cookie 15924778270603038546 [ 1118.947445] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.974086] LustreError: 35657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1119.593621] Lustre: server umount lustre-MDT0001 complete [ 1136.597365] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 1139.472400] Key type lgssc unregistered [ 1139.757254] LNet: 38457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.776070] LNetError: 38457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.843357] LNet: Removed LNI 192.168.202.125@tcp [ 1142.199378] Key type .llcrypt unregistered [ 1142.210764] Key type ._llcrypt unregistered [ 1168.043843] Key type ._llcrypt registered [ 1168.053527] Key type .llcrypt registered [ 1168.170209] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 1169.501755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1169.559302] alg: No test for adler32 (adler32-zlib) [ 1170.889433] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1171.223194] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 1172.983235] Key type lgssc registered [ 1174.836683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1186.171807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1186.194435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.744636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.051311] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1198.057661] LustreError: 39874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1198.093185] LustreError: 39874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1203.176420] LustreError: 39875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.363351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.828780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1209.714311] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1218.698904] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1225.571441] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1234.047852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1234.541469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.868539] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1252.363748] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1258.106210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1264.565352] Lustre: lustre-OST0000: Not available for connect from 192.168.202.25@tcp (stopping) [ 1264.622114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1264.635128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.705144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1268.710959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1268.724085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.761210] Lustre: Skipped 2 previous similar messages [ 1270.485308] Lustre: server umount lustre-OST0000 complete [ 1294.142087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.144286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.150742] Lustre: Skipped 1 previous similar message [ 1294.166273] Lustre: Skipped 2 previous similar messages [ 1298.542896] Lustre: server umount lustre-MDT0000 complete [ 1299.427723] LustreError: 41307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.459553] LustreError: 41307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1302.422327] LustreError: 39855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438733 with bad export cookie 14669582279054056667 [ 1302.425960] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1302.435275] LustreError: 39855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1308.846609] Lustre: server umount lustre-MDT0001 complete [ 1321.796585] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 1325.169229] Key type lgssc unregistered [ 1325.564415] LNet: 43583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.573239] LNetError: 43583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.587395] LNet: Removed LNI 192.168.202.125@tcp [ 1326.520666] Key type .llcrypt unregistered [ 1326.527320] Key type ._llcrypt unregistered [ 1352.538299] Key type ._llcrypt registered [ 1352.546261] Key type .llcrypt registered [ 1352.735787] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 1354.239364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1354.325935] alg: No test for adler32 (adler32-zlib) [ 1356.060930] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1356.538748] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 1358.400462] Key type lgssc registered [ 1361.039395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1372.120899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1372.177838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.928094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.082460] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1384.417702] LustreError: 45005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1384.449369] LustreError: 45005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1389.551223] LustreError: 45004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.514891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1392.058462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1398.112812] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1406.049335] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1413.765665] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1423.543351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1424.155245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.005211] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1441.980581] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1444.607960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1453.546102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1453.550759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.561934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.077803] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.082889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.105589] Lustre: Skipped 1 previous similar message [ 1455.115188] Lustre: Skipped 2 previous similar messages [ 1455.908801] Lustre: server umount lustre-MDT0000 complete [ 1459.080610] LustreError: 45001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1460.431480] LustreError: 44987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438891 with bad export cookie 10032456425578695688 [ 1460.445781] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1464.218750] LustreError: 46444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.256337] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.25@tcp (stopping) [ 1464.256814] LustreError: 46444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1464.274461] Lustre: Skipped 1 previous similar message [ 1465.313169] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.290250] Lustre: server umount lustre-MDT0001 complete [ 1489.581119] Lustre: server umount lustre-OST0000 complete [ 1509.686490] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 1513.549931] Key type lgssc unregistered [ 1513.986185] LNet: 48865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.997464] LNetError: 48865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.049272] LNet: Removed LNI 192.168.202.125@tcp [ 1516.065458] Key type .llcrypt unregistered [ 1516.069829] Key type ._llcrypt unregistered [ 1544.717367] Key type ._llcrypt registered [ 1544.722115] Key type .llcrypt registered [ 1544.905700] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 1546.928606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1546.982123] alg: No test for adler32 (adler32-zlib) [ 1548.207909] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1548.472385] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 1550.287189] Key type lgssc registered [ 1552.027550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1562.258263] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1562.276996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1577.759648] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1582.879434] LustreError: 50251:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.125@tcp: failed processing log, type 4: rc = -110 [ 1609.567896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.600933] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1627.346648] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1792.311592] Lustre: server umount lustre-OST0000 complete [ 1813.423464] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 1817.211563] Key type lgssc unregistered [ 1817.665596] LNet: 51927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.675555] LNetError: 51927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.712546] LNet: Removed LNI 192.168.202.125@tcp [ 1818.734712] Key type .llcrypt unregistered [ 1818.737432] Key type ._llcrypt unregistered [ 1846.897378] Key type ._llcrypt registered [ 1846.900076] Key type .llcrypt registered [ 1847.068171] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 1848.871058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1848.923759] alg: No test for adler32 (adler32-zlib) [ 1850.075237] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1850.369123] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 1852.178050] Key type lgssc registered [ 1853.752439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1863.669956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1863.712968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.211103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.985488] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1875.445261] LustreError: 53337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1875.471061] LustreError: 53337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1880.571797] LustreError: 53338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.539745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.979689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1887.610053] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1897.737748] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1905.161354] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1913.874711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1914.446616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.240378] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 1930.843960] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1934.692750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.695599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.899636] Lustre: server umount lustre-OST0000 complete [ 1945.056915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.079471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.081679] Lustre: Skipped 1 previous similar message [ 1950.176942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1950.187473] Lustre: Skipped 1 previous similar message [ 1951.083709] Lustre: server umount lustre-MDT0000 complete [ 1955.296672] LustreError: 54067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.319486] LustreError: 54067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1955.787090] LustreError: 54074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439387 with bad export cookie 9413041408344672052 [ 1955.805629] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1956.238409] Lustre: server umount lustre-MDT0001 complete [ 1970.529632] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 1974.057752] Key type lgssc unregistered [ 1974.437501] LNet: 56872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.448457] LNetError: 56872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.526927] LNet: Removed LNI 192.168.202.125@tcp [ 1976.331725] Key type .llcrypt unregistered [ 1976.339950] Key type ._llcrypt unregistered [ 2003.717526] Key type ._llcrypt registered [ 2003.722723] Key type .llcrypt registered [ 2003.949790] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 2005.554837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2005.634019] alg: No test for adler32 (adler32-zlib) [ 2007.305917] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2007.551582] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 2009.327193] Key type lgssc registered [ 2010.987276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2022.397559] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2022.440860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2038.111707] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2043.232071] LustreError: 58254:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.125@tcp: failed processing log, type 4: rc = -110 [ 2069.919259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.342105] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2087.809235] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2096.527372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.439810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.512694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2102.756553] LustreError: 58279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2102.785029] LustreError: 58279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2103.532681] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2103.790433] LustreError: 58280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2104.807651] LustreError: 59291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2104.828591] LustreError: 59291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2109.943315] LustreError: 58279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.983042] LustreError: 58279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2114.857639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.045632] LustreError: 59291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.071633] LustreError: 59291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2115.502461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2121.656517] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2131.180746] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2138.052977] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2141.157758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.163031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2141.177764] Lustre: Skipped 1 previous similar message [ 2146.277382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2146.283561] Lustre: Skipped 2 previous similar messages [ 2146.465611] Lustre: server umount lustre-OST0000 complete [ 2171.874639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2171.882276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.897514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2175.969585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2175.972248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.976393] Lustre: server umount lustre-MDT0000 complete [ 2181.088824] LustreError: 59297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.121902] LustreError: 59297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2182.172937] LustreError: 58262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439613 with bad export cookie 6829968926283643700 [ 2182.193627] LustreError: 58262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2182.199947] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2188.593909] Lustre: server umount lustre-MDT0001 complete [ 2203.178175] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 2206.579412] Key type lgssc unregistered [ 2206.982330] LNet: 62003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.989954] LNetError: 62003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.022968] LNet: Removed LNI 192.168.202.125@tcp [ 2208.090074] Key type .llcrypt unregistered [ 2208.096995] Key type ._llcrypt unregistered [ 2237.382602] Key type ._llcrypt registered [ 2237.385889] Key type .llcrypt registered [ 2237.537391] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 2239.244514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2239.578834] alg: No test for adler32 (adler32-zlib) [ 2241.166777] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2241.752577] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 2243.647508] Key type lgssc registered [ 2246.444284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2260.253837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2260.264427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.010528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.477745] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2272.229192] LustreError: 63414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.259432] LustreError: 63414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2277.353480] LustreError: 63415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.317575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.962237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2287.891514] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2297.203677] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2304.350579] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2313.049465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2313.638845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.176770] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2332.090570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2333.477798] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2352.614409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.619333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2352.642662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2357.733198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2357.736877] Lustre: Skipped 2 previous similar messages [ 2358.531655] Lustre: server umount lustre-OST0000 complete [ 2366.952769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2366.973340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.011765] Lustre: Skipped 1 previous similar message [ 2367.021365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2369.957924] Lustre: server umount lustre-MDT0000 complete [ 2373.089806] LustreError: 63410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.120303] LustreError: 63410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2374.694226] LustreError: 63394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439806 with bad export cookie 2265353558675122664 [ 2374.706303] LustreError: 63394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2374.709067] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2375.109632] Lustre: server umount lustre-MDT0001 complete [ 2387.742926] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 2390.363550] Key type lgssc unregistered [ 2390.691909] LNet: 66976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.697112] LNetError: 66976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.724249] LNet: Removed LNI 192.168.202.125@tcp [ 2392.719885] Key type .llcrypt unregistered [ 2392.722879] Key type ._llcrypt unregistered [ 2422.362831] Key type ._llcrypt registered [ 2422.366950] Key type .llcrypt registered [ 2422.508080] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 2423.784702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2424.045702] alg: No test for adler32 (adler32-zlib) [ 2425.213159] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2425.536331] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 2427.215200] Key type lgssc registered [ 2428.464889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2439.607455] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 08:51:08 (1788439868) [ 2447.972675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2447.985849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.444911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.782866] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2459.622368] LustreError: 68677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.671204] LustreError: 68677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2464.409673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.869880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2470.532077] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2480.499863] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2488.108874] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2496.625513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2497.114919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.867897] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2515.136716] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2518.702089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2526.903466] Lustre: Failing over lustre-MDT0000 [ 2527.594372] Lustre: server umount lustre-MDT0000 complete [ 2529.261747] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.278142] LustreError: 71177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.294847] LustreError: 71177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2531.238750] LustreError: 71177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.372189] LustreError: 68677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.395737] LustreError: 68677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2538.056344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.210238] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2538.393441] Lustre: *** cfs_fail_loc=135, val=0*** [ 2538.512078] LustreError: 71770:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2538.531542] LustreError: 71770:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.125@tcp: cfg command failed: rc = -2 [ 2538.545750] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2538.545750] [ 2538.569239] LustreError: MGC192.168.202.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2538.592446] LustreError: 71758:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2538.602918] LustreError: 71758:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2538.614564] LustreError: 71758:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2538.619428] LustreError: 71758:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2538.653509] Lustre: server umount lustre-MDT0000 complete [ 2538.655709] LustreError: 71758:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2539.494350] LustreError: 69407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.523909] LustreError: 69407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2546.395214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.644396] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.653954] LustreError: 68668:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2546.947790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.003665] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2551.700538] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2552.302160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2552.354822] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2552.420356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2552.849700] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2563.040659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2563.048808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.067631] Lustre: Skipped 2 previous similar messages [ 2563.081727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2567.139683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2567.158874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.190818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2568.900557] Lustre: server umount lustre-OST0000 complete [ 2572.770253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.772909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2572.783080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.790153] Lustre: Skipped 2 previous similar messages [ 2577.890808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2577.902214] Lustre: Skipped 1 previous similar message [ 2579.062372] Lustre: server umount lustre-MDT0000 complete [ 2582.920908] LustreError: 71261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440014 with bad export cookie 7782803699354131971 [ 2582.931529] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2583.529471] Lustre: server umount lustre-MDT0001 complete [ 2595.524617] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 2598.667851] Key type lgssc unregistered [ 2598.949773] LNet: 73549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2598.954294] LNetError: 73549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2598.983708] LNet: Removed LNI 192.168.202.125@tcp [ 2600.060781] Key type .llcrypt unregistered [ 2600.063084] Key type ._llcrypt unregistered [ 2627.306173] Key type ._llcrypt registered [ 2627.309159] Key type .llcrypt registered [ 2627.416098] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 2628.418703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2628.459815] alg: No test for adler32 (adler32-zlib) [ 2629.648534] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2630.076500] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 2631.719251] Key type lgssc registered [ 2633.633947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2643.365826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2643.395498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.005662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.644803] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2655.200282] LustreError: 74960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.219911] LustreError: 74960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2660.323315] LustreError: 74959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.745075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.118820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2667.494953] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2676.677718] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2686.988571] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2700.145831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2701.004325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.518619] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2715.335979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2719.621527] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2725.784017] Lustre: Failing over lustre-MDT0000 [ 2725.861974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.864704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2725.881597] Lustre: Skipped 2 previous similar messages [ 2725.894070] Lustre: Skipped 2 previous similar messages [ 2726.063503] Lustre: server umount lustre-MDT0000 complete [ 2726.311241] LustreError: 74954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.978650] LustreError: 75688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.013745] LustreError: 75688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2736.103228] LustreError: 74959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2736.130278] LustreError: 74959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2737.288079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.374517] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2737.492729] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2737.601451] LustreError: 78047:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2737.615453] LustreError: 78047:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.125@tcp: cfg command failed: rc = -9 [ 2737.625868] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2737.625868] [ 2737.636080] LustreError: MGC192.168.202.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2737.656665] LustreError: 78035:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2737.665902] LustreError: 78035:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2737.678129] LustreError: 78035:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2737.687278] LustreError: 78035:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2737.714527] Lustre: server umount lustre-MDT0000 complete [ 2737.717085] LustreError: 78035:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2744.950319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.139940] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.508364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.601657] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2746.757128] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2750.779323] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2750.958172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2751.035586] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2751.092140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2761.193148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.199231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2761.231607] Lustre: Skipped 1 previous similar message [ 2761.258175] Lustre: Skipped 1 previous similar message [ 2766.318099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2766.326555] Lustre: Skipped 1 previous similar message [ 2766.500400] Lustre: server umount lustre-OST0000 complete [ 2771.424128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2771.427716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.435390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2771.455585] Lustre: Skipped 1 previous similar message [ 2776.557320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.572368] Lustre: Skipped 2 previous similar messages [ 2777.973276] Lustre: server umount lustre-MDT0000 complete [ 2781.664348] LustreError: 74960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.696549] LustreError: 74960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2782.608744] LustreError: 74941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440213 with bad export cookie 6230009549107494350 [ 2782.623883] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2783.056360] Lustre: server umount lustre-MDT0001 complete [ 2796.160482] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 2799.300547] Key type lgssc unregistered [ 2799.792430] LNet: 79829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2799.798317] LNetError: 79829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2800.871774] LNet: Removed LNI 192.168.202.125@tcp [ 2802.131860] Key type .llcrypt unregistered [ 2802.134069] Key type ._llcrypt unregistered [ 2819.868277] Key type ._llcrypt registered [ 2819.869483] Key type .llcrypt registered [ 2820.024060] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 2820.398654] Key type .llcrypt unregistered [ 2820.401211] Key type ._llcrypt unregistered [ 2849.281566] Key type ._llcrypt registered [ 2849.284291] Key type .llcrypt registered [ 2849.489607] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 2851.115317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2851.191050] alg: No test for adler32 (adler32-zlib) [ 2852.457681] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2852.704112] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 2854.439682] Key type lgssc registered [ 2856.091507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2865.682690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2865.703964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.081120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2871.805233] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2877.409606] LustreError: 81727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.431778] LustreError: 81727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2881.685165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.189923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2887.361542] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2896.432961] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2903.676636] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2911.980506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2912.233763] LustreError: 83549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.634843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.884878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2920.507331] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 2929.924483] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2942.435825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2942.437832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.453229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2942.469919] Lustre: Skipped 1 previous similar message [ 2942.488944] Lustre: Skipped 1 previous similar message [ 2946.341745] Lustre: server umount lustre-OST0000 complete [ 2952.678206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2952.682535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2952.700119] Lustre: Skipped 1 previous similar message [ 2952.719578] Lustre: Skipped 1 previous similar message [ 2957.486058] Lustre: server umount lustre-MDT0000 complete [ 2957.794084] LustreError: 82455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.820886] LustreError: 82455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2961.679518] LustreError: 81708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440393 with bad export cookie 2336194052985500332 [ 2961.689899] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2961.693565] LustreError: 81708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2962.108600] Lustre: server umount lustre-MDT0001 complete [ 2976.136933] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 2979.651651] Key type lgssc unregistered [ 2980.077731] LNet: 85263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2980.086392] LNetError: 85263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2980.117359] LNet: Removed LNI 192.168.202.125@tcp [ 2981.322591] Key type .llcrypt unregistered [ 2981.325520] Key type ._llcrypt unregistered [ 3009.312360] Key type ._llcrypt registered [ 3009.319455] Key type .llcrypt registered [ 3009.461497] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 3011.172250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3011.208339] alg: No test for adler32 (adler32-zlib) [ 3012.358758] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3012.580576] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 3014.263951] Key type lgssc registered [ 3015.611207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3025.376945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3025.404294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3027.062873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.567167] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3037.161858] LustreError: 86672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.197916] LustreError: 86672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3041.848463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.272575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3046.719356] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3056.750293] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3063.361737] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3070.212496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.576514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.629563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 3077.528390] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3086.641897] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3093.479720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3093.493623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.514525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3097.574807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.584643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3097.600298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3097.603929] Lustre: Skipped 1 previous similar message [ 3099.418512] Lustre: server umount lustre-OST0000 complete [ 3103.712159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3103.716813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.729583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3107.810383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.811419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3107.840800] Lustre: Skipped 1 previous similar message [ 3110.091564] Lustre: server umount lustre-MDT0000 complete [ 3112.931917] LustreError: 87877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.950335] LustreError: 87877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3113.884848] LustreError: 86653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440545 with bad export cookie 7663782053760451967 [ 3113.896018] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3113.903138] LustreError: 86653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3114.294337] Lustre: server umount lustre-MDT0001 complete [ 3125.977839] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 3128.641404] Key type lgssc unregistered [ 3129.093899] LNet: 90219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3129.106406] LNetError: 90219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3130.149069] LNet: Removed LNI 192.168.202.125@tcp [ 3131.600134] Key type .llcrypt unregistered [ 3131.602593] Key type ._llcrypt unregistered [ 3157.687390] Key type ._llcrypt registered [ 3157.693237] Key type .llcrypt registered [ 3157.840835] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 3159.129815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3159.222248] alg: No test for adler32 (adler32-zlib) [ 3160.325166] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3160.587112] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 3162.248401] Key type lgssc registered [ 3163.883501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3172.836630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3172.885401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.654685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.382625] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3184.104441] LustreError: 91632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3184.151865] LustreError: 91632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3189.218226] LustreError: 91633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.520131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.046026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3196.688258] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3207.029743] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3214.137760] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3222.863643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3223.439755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.851607] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3240.059956] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3244.037258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3259.363504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3259.378781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.401040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3264.480298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3264.497089] Lustre: Skipped 2 previous similar messages [ 3265.480159] Lustre: server umount lustre-OST0000 complete [ 3273.190140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3273.209177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.225751] Lustre: Skipped 1 previous similar message [ 3273.242196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3274.756590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3276.976278] Lustre: server umount lustre-MDT0000 complete [ 3279.841255] LustreError: 91628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.865367] LustreError: 91628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3281.827908] LustreError: 91612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440713 with bad export cookie 2287656371788151072 [ 3281.829311] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3281.841935] LustreError: 91612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3282.357760] Lustre: server umount lustre-MDT0001 complete [ 3296.939439] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 3299.683733] Key type lgssc unregistered [ 3299.924924] LNet: 95188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3299.943224] LNetError: 95188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3300.978908] LNet: Removed LNI 192.168.202.125@tcp [ 3301.737424] Key type .llcrypt unregistered [ 3301.739741] Key type ._llcrypt unregistered [ 3331.207826] Key type ._llcrypt registered [ 3331.210793] Key type .llcrypt registered [ 3331.367568] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 3332.839623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3332.867232] alg: No test for adler32 (adler32-zlib) [ 3333.934690] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3334.203321] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 3335.901331] Key type lgssc registered [ 3337.088568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3346.043397] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3346.058489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3361.631631] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3366.751419] LustreError: 96572:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.125@tcp: failed processing log, type 4: rc = -110 [ 3393.375169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3400.542974] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3409.033221] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3415.810342] Lustre: server umount lustre-OST0000 complete [ 3425.683405] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:07:34 (1788440854) [ 3432.699075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.038275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3438.060684] LustreError: 98211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.088316] LustreError: 98211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3438.690320] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3443.178388] LustreError: 98212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.295192] LustreError: 98212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.452235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.751582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3453.865595] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3462.806755] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3472.078848] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3480.646886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3481.028245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.969424] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3495.124841] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3503.552424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3518.945948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3518.961575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.968624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3518.975273] Lustre: Skipped 1 previous similar message [ 3518.998323] Lustre: Skipped 1 previous similar message [ 3524.072792] LustreError: 100035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3524.091907] LustreError: 100035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3524.297385] Lustre: server umount lustre-OST0000 complete [ 3529.191181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.195519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.204293] Lustre: Skipped 1 previous similar message [ 3534.308730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3534.314977] Lustre: Skipped 2 previous similar messages [ 3535.194859] Lustre: server umount lustre-MDT0000 complete [ 3539.435939] LustreError: 99197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3539.461986] LustreError: 99197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3539.542120] LustreError: 98193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440970 with bad export cookie 14690116387061540626 [ 3539.544555] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3539.551562] LustreError: 98193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3539.906502] Lustre: server umount lustre-MDT0001 complete [ 3553.687669] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 3557.412846] Key type lgssc unregistered [ 3557.740617] LNet: 101928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3557.746829] LNetError: 101928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3557.759414] LNet: Removed LNI 192.168.202.125@tcp [ 3558.623528] Key type .llcrypt unregistered [ 3558.625487] Key type ._llcrypt unregistered [ 3606.448620] Key type ._llcrypt registered [ 3606.450682] Key type .llcrypt registered [ 3606.580772] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 3608.588584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3608.621218] alg: No test for adler32 (adler32-zlib) [ 3609.881263] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3610.252758] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 3612.099220] Key type lgssc registered [ 3613.218763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3622.866521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3622.914717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3624.637150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3630.313025] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3634.146480] LustreError: 104111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3634.171685] LustreError: 104111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3639.273908] LustreError: 104112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.221161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.631629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3648.355618] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3657.318520] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3664.247288] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3671.129843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3671.545810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3679.788811] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3691.029664] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3694.075556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3709.410868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3709.422496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3709.439559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3709.448077] Lustre: Skipped 1 previous similar message [ 3710.334275] Lustre: server umount lustre-OST0000 complete [ 3719.657916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.677766] Lustre: Skipped 1 previous similar message [ 3719.686707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3719.687146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3721.834425] Lustre: server umount lustre-MDT0000 complete [ 3724.768917] LustreError: 104108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3724.786573] LustreError: 104108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3726.647427] LustreError: 104092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441157 with bad export cookie 1493424175634025901 [ 3726.654814] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3726.665058] LustreError: 104092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3726.997995] Lustre: server umount lustre-MDT0001 complete [ 3742.098162] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 3745.448379] Key type lgssc unregistered [ 3745.819837] LNet: 107668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3745.837312] LNetError: 107668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3745.877620] LNet: Removed LNI 192.168.202.125@tcp [ 3747.113369] Key type .llcrypt unregistered [ 3747.118234] Key type ._llcrypt unregistered [ 3766.907377] Key type ._llcrypt registered [ 3766.919204] Key type .llcrypt registered [ 3767.067279] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 3768.890721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3768.940805] alg: No test for adler32 (adler32-zlib) [ 3770.097416] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3770.292884] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 3771.920058] Key type lgssc registered [ 3773.074728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3781.365848] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3781.384383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3796.961388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3802.079451] LustreError: 108920:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.125@tcp: failed processing log, type 4: rc = -110 [ 3828.706755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.420123] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3846.184395] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3854.407294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3854.931796] LustreError: MGC192.168.202.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3854.949371] LustreError: 109944:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3854.957466] LustreError: 109944:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3854.967839] LustreError: 109944:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3855.040685] Lustre: server umount lustre-MDT0000 complete [ 3855.043491] LustreError: 109944:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3872.735121] Lustre: 108256:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441288/real 1788441288] req@00000000db24c0f2 x1875316520783232/t0(0) o400->MGC192.168.202.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788441304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3872.777194] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3872.790334] LustreError: 108932:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3873.029261] Lustre: server umount lustre-OST0000 complete [ 3884.058693] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 3894.401724] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 3938.430980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.660837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3938.681450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3938.745720] Lustre: lustre-MDT0000: new disk, initializing [ 3938.816104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3938.834727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3944.569945] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3957.515083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3957.639850] Lustre: 112739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3957.727712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3957.729826] Lustre: Skipped 1 previous similar message [ 3957.817303] Lustre: lustre-MDT0001: new disk, initializing [ 3957.904156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3957.955472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3957.972980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3963.507188] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 3973.983797] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3980.414258] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3988.358649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3988.774365] Lustre: lustre-OST0000: new disk, initializing [ 3988.780474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3988.792945] Lustre: 114712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3988.904217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3996.752839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3996.763051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3996.854575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3996.891697] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4008.224973] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4013.584650] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4013.891865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4019.443964] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4019.742624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4022.753105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4022.767940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4022.784204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4022.801048] Lustre: Skipped 1 previous similar message [ 4027.872775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4027.885225] Lustre: Skipped 1 previous similar message [ 4028.442315] Lustre: server umount lustre-OST0000 complete [ 4032.992535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4032.999533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4033.000327] Lustre: Skipped 2 previous similar messages [ 4033.009818] Lustre: Skipped 1 previous similar message [ 4038.117315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4038.131454] Lustre: Skipped 1 previous similar message [ 4038.749905] Lustre: server umount lustre-MDT0000 complete [ 4043.262980] LustreError: 113534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441474 with bad export cookie 6078928446632036705 [ 4043.267686] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4043.277527] LustreError: 113534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4043.674160] Lustre: server umount lustre-MDT0001 complete [ 4054.078610] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:18:03 (1788441483) [ 4058.161159] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4077.887913] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 4087.368813] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 4128.489404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4128.853442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4128.910846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4129.096175] Lustre: lustre-MDT0000: new disk, initializing [ 4129.283823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.314623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4136.772784] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4148.692207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.972447] Lustre: 119613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4149.119823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4149.128550] Lustre: Skipped 1 previous similar message [ 4149.309337] Lustre: lustre-MDT0001: new disk, initializing [ 4149.414818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4149.457817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4149.478328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4153.658558] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4163.976900] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4170.410701] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4178.057603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4178.476743] Lustre: lustre-OST0000: new disk, initializing [ 4178.481156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4178.489548] Lustre: 121581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4178.583775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.190899] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4187.780773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4187.796833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4187.970313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4197.657687] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4203.127980] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4203.344174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4207.876783] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4208.104773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4213.736793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4213.738712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.743558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4213.760366] Lustre: Skipped 1 previous similar message [ 4216.143994] Lustre: server umount lustre-OST0000 complete [ 4223.978890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.994276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4224.000492] Lustre: Skipped 1 previous similar message [ 4224.008240] Lustre: Skipped 1 previous similar message [ 4236.255230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4236.564078] Lustre: server umount lustre-MDT0000 complete [ 4239.352371] LustreError: 119624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4239.375608] LustreError: 119624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4240.686496] LustreError: 119604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441672 with bad export cookie 6078928446632037587 [ 4240.700200] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4241.040897] Lustre: server umount lustre-MDT0001 complete [ 4250.648059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4251.120538] LustreError: 123808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4251.237285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4256.233932] LustreError: 123809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4257.114411] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4261.350579] LustreError: 123808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.464180] LustreError: 123809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4267.672675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.539889] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4282.741490] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4290.313045] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4298.364307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4307.054813] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4317.930724] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4329.474861] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4337.121504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4337.140075] LustreError: Skipped 1 previous similar message [ 4337.149755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.152695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4337.152702] Lustre: Skipped 5 previous similar messages [ 4337.185935] Lustre: Skipped 1 previous similar message [ 4341.281728] Lustre: server umount lustre-OST0000 complete [ 4347.361537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.384646] Lustre: Skipped 1 previous similar message [ 4360.160125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4360.402709] Lustre: server umount lustre-MDT0000 complete [ 4362.723040] LustreError: 123808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4362.743764] LustreError: 123808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4365.072273] LustreError: 126688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441796 with bad export cookie 6078928446632038469 [ 4365.079761] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4365.088772] LustreError: 126688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4365.586660] Lustre: server umount lustre-MDT0001 complete [ 4379.128345] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 4382.979805] Key type lgssc unregistered [ 4383.457826] LNet: 127442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4383.476114] LNetError: 127442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4383.506335] LNet: Removed LNI 192.168.202.125@tcp [ 4384.593790] Key type .llcrypt unregistered [ 4384.596773] Key type ._llcrypt unregistered [ 4406.760540] Key type ._llcrypt registered [ 4406.762817] Key type .llcrypt registered [ 4406.873737] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 4425.313797] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 4426.558653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4426.926584] alg: No test for adler32 (adler32-zlib) [ 4428.223653] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4428.592990] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 4430.375976] Key type lgssc registered [ 4431.815757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4478.333403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4478.362890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.730349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4479.798847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4479.883169] Lustre: lustre-MDT0000: new disk, initializing [ 4479.952835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.970775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4485.246777] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4497.647526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4497.766353] Lustre: 130595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4497.817422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4497.822723] Lustre: Skipped 1 previous similar message [ 4497.901996] Lustre: lustre-MDT0001: new disk, initializing [ 4498.000492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4498.041257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4498.070349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4503.972151] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4515.327541] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4521.973585] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4530.938220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4531.168802] Lustre: lustre-OST0000: new disk, initializing [ 4531.172254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4531.177680] Lustre: 132563:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4531.259726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4537.433290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4537.452391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4537.573250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4538.663115] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4549.782951] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4555.449525] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4555.663906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4560.098684] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4560.345664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4563.433827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.437528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4563.438116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4563.455931] Lustre: Skipped 1 previous similar message [ 4563.490920] LustreError: Skipped 1 previous similar message [ 4568.686126] Lustre: server umount lustre-OST0000 complete [ 4573.671260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.698631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4573.714452] Lustre: Skipped 2 previous similar messages [ 4578.786869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4578.793809] Lustre: Skipped 1 previous similar message [ 4579.041971] Lustre: server umount lustre-MDT0000 complete [ 4583.615043] LustreError: 133905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442014 with bad export cookie 4759499973715232913 [ 4583.621231] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.624652] LustreError: 133905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4584.201273] Lustre: server umount lustre-MDT0001 complete [ 4593.743457] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:27:02 (1788442022) [ 4601.649959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4602.373286] LustreError: 134978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.392193] LustreError: 134978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4602.434410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4607.460713] LustreError: 134979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4607.544099] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4612.579830] LustreError: 134978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4616.372604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.823445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4621.701323] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4630.139303] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4637.699828] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4642.273484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4642.297753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4642.323219] Lustre: Skipped 1 previous similar message [ 4642.335837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4646.924294] Lustre: server umount lustre-MDT0000 complete [ 4648.935268] LustreError: 134979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.950953] LustreError: 134979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4650.783683] LustreError: 134961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442082 with bad export cookie 4759499973715233795 [ 4650.791484] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4650.798908] LustreError: 134961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4651.294852] Lustre: server umount lustre-MDT0001 complete [ 4662.298823] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:28:11 (1788442091) [ 4670.187797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4685.856556] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4690.975726] LustreError: 137396:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.125@tcp: failed processing log, type 4: rc = -110 [ 4716.639400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4723.542261] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4732.234192] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4734.711799] Lustre: server umount lustre-OST0000 complete [ 4743.390482] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:29:32 (1788442172) [ 4750.700740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.131290] LustreError: 138843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.227696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4756.153451] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4761.577436] LustreError: 138843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.601353] LustreError: 138843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4765.268560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.618316] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4779.742179] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4786.082851] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4794.446095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4794.774333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4794.785692] Lustre: Skipped 1 previous similar message [ 4802.509247] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4810.874788] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4820.451850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4820.474278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4820.480743] Lustre: Skipped 2 previous similar messages [ 4820.516144] Lustre: Skipped 5 previous similar messages [ 4824.935951] Lustre: server umount lustre-MDT0000 complete [ 4825.569996] LustreError: 138844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4825.605217] LustreError: 138844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4829.795467] LustreError: 138824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442261 with bad export cookie 4759499973715234460 [ 4829.802804] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4829.811649] LustreError: 138824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4830.169905] Lustre: server umount lustre-MDT0001 complete [ 4845.592248] Lustre: server umount lustre-OST0000 complete [ 4855.100626] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:31:24 (1788442284) [ 4863.329320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.821307] LustreError: 142497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4863.920455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4869.432987] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4878.707780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4883.110921] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4892.639977] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4899.394905] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4907.086315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4909.575034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4915.194820] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 4924.298806] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4930.385298] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4930.666897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4935.942745] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4936.189049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4939.233838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4939.244789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.253147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4939.270137] Lustre: Skipped 2 previous similar messages [ 4939.285333] Lustre: Skipped 1 previous similar message [ 4944.898844] Lustre: server umount lustre-OST0000 complete [ 4951.010129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4951.022724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4956.136642] Lustre: server umount lustre-MDT0000 complete [ 4959.717300] LustreError: 142497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.736575] LustreError: 142497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4960.414440] LustreError: 142479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442391 with bad export cookie 4759499973715235531 [ 4960.427402] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4960.841458] Lustre: server umount lustre-MDT0001 complete [ 4971.029379] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:33:20 (1788442400) [ 4979.297843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4995.103882] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5000.223635] LustreError: 146659:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.125@tcp: failed processing log, type 4: rc = -110 [ 5025.953737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5025.964495] Lustre: Skipped 2 previous similar messages [ 5033.691791] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5043.411454] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5051.131580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.794319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5056.678231] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5065.899087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5070.416948] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5078.373944] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5084.768812] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5090.197668] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5090.456992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5094.713553] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5094.910229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5096.933534] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.935208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5096.936479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.936485] Lustre: Skipped 5 previous similar messages [ 5096.946431] Lustre: Skipped 2 previous similar messages [ 5103.038318] Lustre: server umount lustre-MDT0000 complete [ 5107.058813] LustreError: 146665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442538 with bad export cookie 4759499973715236553 [ 5107.068549] LustreError: 146665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5107.072726] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5107.173343] LustreError: 147697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5107.189158] LustreError: 147697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5107.390827] Lustre: server umount lustre-MDT0001 complete [ 5113.217175] Lustre: server umount lustre-OST0000 complete [ 5121.183313] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:35:50 (1788442550) [ 5128.274025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5143.839673] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5148.959930] LustreError: 150852:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.125@tcp: failed processing log, type 4: rc = -110 [ 5181.275784] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5189.457917] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5197.453341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5198.122337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5203.486368] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5213.177671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.858113] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5227.480460] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5234.383163] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5241.510215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5241.682713] Lustre: lustre-OST0001: new disk, initializing [ 5241.687319] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5241.691444] Lustre: 153726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5247.533935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5247.547936] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5247.656380] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5248.347271] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5258.386943] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5264.606902] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5265.062543] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5270.626638] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5270.818544] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5273.062934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.074090] Lustre: Skipped 3 previous similar messages [ 5273.080486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5273.085359] Lustre: Skipped 7 previous similar messages [ 5278.927299] Lustre: server umount lustre-OST0000 complete [ 5289.537035] Lustre: server umount lustre-OST0001 complete [ 5299.971249] Lustre: server umount lustre-MDT0000 complete [ 5304.097827] LustreError: 150859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442735 with bad export cookie 4759499973715237505 [ 5304.105858] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5304.109496] LustreError: 150859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5304.401449] Lustre: server umount lustre-MDT0001 complete [ 5330.326358] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:39:19 (1788442759) [ 5332.177948] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5334.165715] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5336.042715] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:39:25 (1788442765) [ 5343.090040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5343.537846] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5343.557791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5343.728947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5343.734586] Lustre: Skipped 6 previous similar messages [ 5349.034491] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5358.405980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.576266] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5358.633792] Lustre: 157288:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5364.085633] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5372.555470] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5379.141452] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5386.860534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5387.159211] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5394.102970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5394.375717] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5402.745861] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5408.462360] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5408.664752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5414.799155] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5415.169392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5420.003507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5420.017299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5420.017438] LustreError: Skipped 1 previous similar message [ 5420.027563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5420.045416] Lustre: Skipped 5 previous similar messages [ 5420.070105] Lustre: Skipped 12 previous similar messages [ 5423.628470] Lustre: server umount lustre-OST0000 complete [ 5447.718296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5454.929364] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5460.587876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.603825] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5463.106195] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5468.651023] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5468.790565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5470.830396] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.850357] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5473.607718] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5473.825874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5480.833423] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff910446d25000.ost_server_uuid 50 [ 5482.631739] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff910446d25000.ost_server_uuid in FULL state after 0 sec [ 5482.845343] LustreError: 157295:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.25@tcp) returned error from blocking AST (req@0000000019758e69 x1875317211013632 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000008346de08/0x420d240259c723db lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.25@tcp remote: 0x893e72a5fffc9296 expref: 5 pid: 157295 timeout: 5582 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5482.900749] LustreError: lustre-MDT0000: A client on nid 192.168.202.25@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5482.917775] LustreError: 157283:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.25@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000008346de08/0x420d240259c723db lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.25@tcp remote: 0x893e72a5fffc9296 expref: 6 pid: 157295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5489.124948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5494.348539] Lustre: server umount lustre-OST0000 complete [ 5504.498575] Lustre: server umount lustre-MDT0000 complete [ 5507.045077] LustreError: 157295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5507.059254] LustreError: 157295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 5508.366602] LustreError: 157279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442939 with bad export cookie 4759499973715238688 [ 5508.366990] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5508.373408] LustreError: 157279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5514.749512] Lustre: server umount lustre-MDT0001 complete [ 5527.382645] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 5530.293715] Key type lgssc unregistered [ 5530.554153] LNet: 163406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5530.565443] LNetError: 163406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5531.625510] LNet: Removed LNI 192.168.202.125@tcp [ 5532.484682] Key type .llcrypt unregistered [ 5532.487063] Key type ._llcrypt unregistered [ 5558.756555] Key type ._llcrypt registered [ 5558.759814] Key type .llcrypt registered [ 5558.913795] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 5560.306238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5560.368954] alg: No test for adler32 (adler32-zlib) [ 5561.443234] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5561.676891] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 5563.343202] Key type lgssc registered [ 5564.360425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5572.542536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5572.568214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5574.004450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5577.596911] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5584.359233] LustreError: 164818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5584.387722] LustreError: 164818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5586.105962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5586.449829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5591.645546] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5601.103117] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5608.212708] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5616.511293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5617.134912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5619.200168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5624.506627] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5633.148293] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5640.762971] Lustre: Failing over lustre-MDT0000 [ 5641.130992] LustreError: 164813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.202425] Lustre: server umount lustre-MDT0000 complete [ 5641.709575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5641.729442] Lustre: Skipped 1 previous similar message [ 5646.817374] LustreError: 164814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5646.849048] LustreError: 164814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5651.940961] LustreError: 165547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.974403] LustreError: 165547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5653.387758] LustreError: 164798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.25@tcp arrived at 1788443084 with bad export cookie 18087525467701464886 [ 5653.403275] LustreError: 164798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5657.069146] LustreError: 164818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5657.099411] LustreError: 164818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5658.080177] Lustre: 164126:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443073/real 1788443073] req@00000000d566a078 x1875318399857024/t0(0) o400->MGC192.168.202.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788443089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5658.107882] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5662.712734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.328113] LustreError: 165547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5668.333832] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfb03cc94c02d4f98 [ 5668.344169] LustreError: 165547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5668.349635] Lustre: MGC192.168.202.125@tcp: Connection restored to 0@lo (at 0@lo) [ 5668.562906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5668.606799] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5673.127531] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5673.980470] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5673.995037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5675.722392] Lustre: lustre-MDT0000: Denying connection for new client a7d743d9-a4e0-4e09-875b-a23b67269ce5 (at 192.168.202.25@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5684.200316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5684.210777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5684.217480] Lustre: Skipped 2 previous similar messages [ 5689.314698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5689.329337] Lustre: Skipped 2 previous similar messages [ 5690.509435] Lustre: server umount lustre-OST0000 complete [ 5694.952656] LustreError: 168550:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5694.968783] Lustre: 167898:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5694.977219] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5695.011111] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5695.027246] LustreError: 167898:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000014571893 x1875318399877120/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5695.037858] LustreError: 167898:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5695.043435] LustreError: 167898:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5695.050656] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5695.562185] Lustre: server umount lustre-MDT0000 complete [ 5699.567953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.607556] LustreError: 164814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5700.005696] LustreError: 164800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443131 with bad export cookie 18087525467701464984 [ 5700.014969] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5700.335482] Lustre: server umount lustre-MDT0001 complete [ 5713.477603] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 5717.063546] Key type lgssc unregistered [ 5717.420347] LNet: 169303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5717.424893] LNetError: 169303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5717.435847] LNet: Removed LNI 192.168.202.125@tcp [ 5718.576366] Key type .llcrypt unregistered [ 5718.581296] Key type ._llcrypt unregistered [ 5747.315804] Key type ._llcrypt registered [ 5747.317786] Key type .llcrypt registered [ 5747.435208] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 5748.372987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5748.420220] alg: No test for adler32 (adler32-zlib) [ 5749.606244] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5749.883538] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 5751.640382] Key type lgssc registered [ 5752.890151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5765.930688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5765.969757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5767.787799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5773.699080] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5778.407908] LustreError: 170713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.431770] LustreError: 170713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5783.367628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.520782] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5783.687435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5787.802550] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5796.631426] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5803.365769] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5811.399063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5811.728895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5812.790202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5818.768210] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5829.010282] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5838.314870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5838.321709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5838.325257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5838.353919] Lustre: Skipped 1 previous similar message [ 5843.433306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5843.452896] Lustre: Skipped 2 previous similar messages [ 5844.358626] Lustre: server umount lustre-OST0000 complete [ 5848.562716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5848.591082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5848.604459] Lustre: Skipped 1 previous similar message [ 5853.666061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5853.675072] Lustre: Skipped 1 previous similar message [ 5862.367153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5862.579566] Lustre: server umount lustre-MDT0000 complete [ 5863.904789] LustreError: 171761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.919605] LustreError: 171761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5865.796424] LustreError: 170695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443297 with bad export cookie 15506622663342060569 [ 5865.813853] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5866.044178] Lustre: server umount lustre-MDT0001 complete [ 5881.603205] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 5885.218176] Key type lgssc unregistered [ 5885.653740] LNet: 174247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5885.677314] LNetError: 174247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5886.772504] LNet: Removed LNI 192.168.202.125@tcp [ 5887.622244] Key type .llcrypt unregistered [ 5887.624333] Key type ._llcrypt unregistered [ 5925.113579] Key type ._llcrypt registered [ 5925.116422] Key type .llcrypt registered [ 5925.231785] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 5926.784976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5926.811603] alg: No test for adler32 (adler32-zlib) [ 5927.925950] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5928.191032] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 5929.871179] Key type lgssc registered [ 5931.447639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5940.822618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5940.836969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5942.281799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5946.922518] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5952.493982] LustreError: 175947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.519604] LustreError: 175947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5956.636282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5957.104368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5962.920773] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 5972.129586] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5978.904962] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5986.947458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5987.375291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5995.653741] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6005.673560] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6011.961363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6027.233404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6027.236728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6027.244089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6027.258183] Lustre: Skipped 1 previous similar message [ 6029.968395] Lustre: server umount lustre-OST0000 complete [ 6037.478241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.479793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6037.493649] Lustre: Skipped 1 previous similar message [ 6037.518566] Lustre: Skipped 2 previous similar messages [ 6040.491041] Lustre: server umount lustre-MDT0000 complete [ 6042.593388] LustreError: 175944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.614775] LustreError: 175944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6044.916638] LustreError: 175928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443476 with bad export cookie 13473526466726298683 [ 6044.932558] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6045.588869] Lustre: server umount lustre-MDT0001 complete [ 6058.224181] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 6061.007715] Key type lgssc unregistered [ 6061.297284] LNet: 179505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6061.303568] LNetError: 179505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6061.316199] LNet: Removed LNI 192.168.202.125@tcp [ 6062.197163] Key type .llcrypt unregistered [ 6062.201462] Key type ._llcrypt unregistered [ 6090.284136] Key type ._llcrypt registered [ 6090.285651] Key type .llcrypt registered [ 6090.414027] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 6091.140051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6091.290171] alg: No test for adler32 (adler32-zlib) [ 6092.433914] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6092.682536] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 6094.383153] Key type lgssc registered [ 6095.151853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6106.445204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6106.466869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.851447] Lustre: *** cfs_fail_loc=135, val=0*** [ 6107.981925] LustreError: 181024:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6107.987868] LustreError: 181024:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.125@tcp: cfg command failed: rc = -2 [ 6107.994869] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6107.994869] [ 6108.010869] LustreError: MGC192.168.202.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6108.022792] LustreError: 180986:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6108.036515] LustreError: 180986:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 6108.048927] LustreError: 180986:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 6108.055172] LustreError: 180986:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6108.176430] Lustre: server umount lustre-MDT0000 complete [ 6108.179225] LustreError: 180986:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6114.366497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.700783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6118.860718] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6125.034257] LustreError: 181396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.060519] LustreError: 181396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6128.644563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6128.905639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6133.239556] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6144.113516] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6151.643212] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6154.732164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6154.753277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6154.784701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6155.744649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6155.745073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.834311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6156.844370] Lustre: Skipped 2 previous similar messages [ 6160.504520] Lustre: server umount lustre-MDT0000 complete [ 6161.890420] LustreError: 181396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.904495] LustreError: 181396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6164.518163] LustreError: 181375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443595 with bad export cookie 9989433963962190367 [ 6164.549359] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6164.991084] Lustre: server umount lustre-MDT0001 complete [ 6179.092441] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 6181.513968] Key type lgssc unregistered [ 6181.773718] LNet: 183716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6181.781347] LNetError: 183716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6181.802791] LNet: Removed LNI 192.168.202.125@tcp [ 6182.645642] Key type .llcrypt unregistered [ 6182.647054] Key type ._llcrypt unregistered [ 6210.954759] Key type ._llcrypt registered [ 6210.956248] Key type .llcrypt registered [ 6211.031483] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 6211.325664] Key type .llcrypt unregistered [ 6211.328392] Key type ._llcrypt unregistered [ 6227.773337] Key type ._llcrypt registered [ 6227.775585] Key type .llcrypt registered [ 6227.877576] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 6229.025575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6229.158638] alg: No test for adler32 (adler32-zlib) [ 6230.264478] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6230.633693] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 6232.415204] Key type lgssc registered [ 6233.429488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6241.243604] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6241.260463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6256.866764] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6261.983349] LustreError: 185842:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.125@tcp: failed processing log, type 4: rc = -110 [ 6288.607185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6298.366936] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6309.715557] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6318.279212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6319.160231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6319.278642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6327.905442] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6328.812683] LustreError: 185867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6328.858994] LustreError: 185867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6333.922931] LustreError: 186881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6333.965035] LustreError: 186881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6339.043493] LustreError: 187016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6339.064262] LustreError: 187016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6341.537709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.847329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6346.848798] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6355.544483] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6362.657119] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6365.882564] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6380.513092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6380.519508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6380.531743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6383.074410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6383.076555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6383.099238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6388.199577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6388.202638] Lustre: Skipped 1 previous similar message [ 6393.312919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6393.323767] Lustre: Skipped 1 previous similar message [ 6393.323965] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 6393.624661] Lustre: server umount lustre-OST0000 complete [ 6403.553667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.566914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6403.571273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6403.574261] Lustre: Skipped 2 previous similar messages [ 6413.799739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6413.813435] Lustre: Skipped 3 previous similar messages [ 6414.307050] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6414.669174] Lustre: server umount lustre-MDT0000 complete [ 6418.914900] LustreError: 186880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6418.951516] LustreError: 186880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6420.644693] LustreError: 185848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443851 with bad export cookie 4320004124164969646 [ 6420.656701] LustreError: 185848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6420.659448] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6421.164943] Lustre: server umount lustre-MDT0001 complete [ 6433.462899] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 6436.411128] Key type lgssc unregistered [ 6436.811600] LNet: 189861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6436.818330] LNetError: 189861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6436.862360] LNet: Removed LNI 192.168.202.125@tcp [ 6437.728445] Key type .llcrypt unregistered [ 6437.732146] Key type ._llcrypt unregistered [ 6465.997742] Key type ._llcrypt registered [ 6466.001939] Key type .llcrypt registered [ 6466.171622] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 6467.867302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6467.887636] alg: No test for adler32 (adler32-zlib) [ 6469.024275] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6469.310570] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 6470.983155] Key type lgssc registered [ 6472.016532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6481.824713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6481.859730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.202961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6487.062460] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6493.670180] LustreError: 191270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6493.691581] LustreError: 191270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6498.287388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6498.685618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6505.009220] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6515.010459] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6521.804955] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6529.904700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6530.228925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6537.651910] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6546.546247] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6553.845507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6558.126590] Lustre: Failing over lustre-MDT0000 [ 6558.440803] Lustre: server umount lustre-MDT0000 complete [ 6559.152803] LustreError: 191267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.209753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6559.219652] Lustre: Skipped 1 previous similar message [ 6564.229723] LustreError: 191266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6564.251857] LustreError: 191266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6569.352074] LustreError: 191267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6569.389547] LustreError: 191267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6574.475957] LustreError: 191265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6574.514236] LustreError: 191265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6575.583115] Lustre: 190581:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443990/real 1788443990] req@0000000094dd5afe x1875319350915584/t0(0) o400->MGC192.168.202.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788444006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6575.631648] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6578.629913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.723284] LustreError: 191265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6584.758135] LustreError: 191265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6585.824577] LustreError: 190580:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006cdd32b3 x1875319350921216/t0(0) o250->MGC192.168.202.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6586.145048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6586.207716] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6589.832642] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6591.367940] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6591.480454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6591.499683] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6591.558695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6596.531799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6602.936482] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6617.060949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6617.061871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6617.063868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6617.068897] Lustre: Skipped 2 previous similar messages [ 6622.196483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6622.207669] Lustre: Skipped 2 previous similar messages [ 6627.309349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6627.320205] Lustre: Skipped 1 previous similar message [ 6631.391364] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6631.938642] Lustre: server umount lustre-OST0000 complete [ 6637.539605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6637.540600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6637.564802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6642.657399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6642.676055] Lustre: Skipped 2 previous similar messages [ 6643.471258] Lustre: server umount lustre-MDT0000 complete [ 6647.793834] LustreError: 191270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6647.820365] LustreError: 191270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6648.227912] LustreError: 191254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444079 with bad export cookie 3709686931849585786 [ 6648.231320] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6648.244434] LustreError: 191254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6648.608682] Lustre: server umount lustre-MDT0001 complete [ 6661.490789] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 6666.005080] Key type lgssc unregistered [ 6666.495814] LNet: 196147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6666.506688] LNetError: 196147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6666.533872] LNet: Removed LNI 192.168.202.125@tcp [ 6667.737707] Key type .llcrypt unregistered [ 6667.740115] Key type ._llcrypt unregistered [ 6696.111824] Key type ._llcrypt registered [ 6696.113049] Key type .llcrypt registered [ 6696.272793] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 6697.026685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6697.053025] alg: No test for adler32 (adler32-zlib) [ 6698.504719] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6698.915597] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 6700.671343] Key type lgssc registered [ 6701.504323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6709.998194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6710.021792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.587914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6716.239703] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6721.004215] LustreError: 197557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6721.027447] LustreError: 197557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6724.875234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6725.198516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6729.055599] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6736.885995] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6743.513132] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6751.322634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6751.814323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6755.971782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6758.850802] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6767.480805] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6771.905423] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6774.660950] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6793.899580] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6811.621335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6811.631234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6811.648877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6811.669039] Lustre: Skipped 1 previous similar message [ 6812.663307] Lustre: server umount lustre-OST0000 complete [ 6821.870266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6821.886892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6821.890352] Lustre: Skipped 1 previous similar message [ 6821.902728] Lustre: Skipped 2 previous similar messages [ 6823.946628] Lustre: server umount lustre-MDT0000 complete [ 6826.976894] LustreError: 197556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6826.991334] LustreError: 197556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6827.696484] LustreError: 197537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444259 with bad export cookie 15692861082282600822 [ 6827.702103] LustreError: 197537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6827.702275] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6828.132726] Lustre: server umount lustre-MDT0001 complete [ 6838.884071] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 6841.953408] Key type lgssc unregistered [ 6842.304437] LNet: 201450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6842.316827] LNetError: 201450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6842.340380] LNet: Removed LNI 192.168.202.125@tcp [ 6843.390319] Key type .llcrypt unregistered [ 6843.407073] Key type ._llcrypt unregistered [ 6864.972083] Key type ._llcrypt registered [ 6864.974174] Key type .llcrypt registered [ 6865.067535] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 6866.075230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6866.190098] alg: No test for adler32 (adler32-zlib) [ 6867.378638] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6867.731107] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 6869.415183] Key type lgssc registered [ 6870.270121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6875.935236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6875.952383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.259862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6881.136180] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6887.408976] LustreError: 202709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6887.441909] LustreError: 202709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6889.952661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6890.415025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6895.714669] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6903.594472] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6912.296647] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6918.332644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6918.727942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.791226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6925.561285] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 6934.348730] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6942.714995] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6953.786103] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6967.202880] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6971.707349] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6983.136461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6983.140901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.150593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6986.211119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6986.216343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6986.242462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.574579] Lustre: server umount lustre-OST0000 complete [ 6992.865141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6992.874250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6992.891468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6992.896585] Lustre: Skipped 1 previous similar message [ 6996.454070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6996.454922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6996.477542] Lustre: Skipped 1 previous similar message [ 6997.908778] Lustre: server umount lustre-MDT0000 complete [ 7001.244759] LustreError: 202689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444432 with bad export cookie 8826776129761220430 [ 7001.258669] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7001.706099] Lustre: server umount lustre-MDT0001 complete [ 7012.540355] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 7015.491342] Key type lgssc unregistered [ 7015.947242] LNet: 207580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7015.951925] LNetError: 207580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7015.974705] LNet: Removed LNI 192.168.202.125@tcp [ 7016.652683] Key type .llcrypt unregistered [ 7016.654287] Key type ._llcrypt unregistered [ 7046.483213] Key type ._llcrypt registered [ 7046.484577] Key type .llcrypt registered [ 7046.688972] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 7048.613163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7048.641136] alg: No test for adler32 (adler32-zlib) [ 7049.854569] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7050.254931] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 7052.047190] Key type lgssc registered [ 7053.235868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7062.784898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7062.800492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7064.174145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7068.730492] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7073.762657] LustreError: 208990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.799306] LustreError: 208990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7077.725056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.095414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7083.377598] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7091.857508] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7098.017478] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7105.202627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7105.638528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7108.746474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7111.495663] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7119.687363] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7134.697771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7134.721941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.741973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7138.787248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.787758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7140.727123] Lustre: server umount lustre-OST0000 complete [ 7149.027769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7149.038553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7149.041425] Lustre: Skipped 1 previous similar message [ 7152.035249] Lustre: server umount lustre-MDT0000 complete [ 7154.149920] LustreError: 208990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.188304] LustreError: 208990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7157.060401] LustreError: 208971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444588 with bad export cookie 9982452225421828728 [ 7157.085146] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7157.788839] Lustre: server umount lustre-MDT0001 complete [ 7169.352611] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 7172.362890] Key type lgssc unregistered [ 7172.637203] LNet: 212675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7172.641965] LNetError: 212675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7172.662721] LNet: Removed LNI 192.168.202.125@tcp [ 7173.522676] Key type .llcrypt unregistered [ 7173.524621] Key type ._llcrypt unregistered [ 7191.914201] Key type ._llcrypt registered [ 7191.916177] Key type .llcrypt registered [ 7191.994620] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 7192.877816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7192.895561] alg: No test for adler32 (adler32-zlib) [ 7194.108678] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7194.390095] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 7196.088137] Key type lgssc registered [ 7197.547310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7206.442451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7206.465065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7208.157102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7213.457048] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7218.664630] LustreError: 213919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.688459] LustreError: 213919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7222.922852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7223.326859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7228.206928] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7234.819692] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7240.776516] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7247.697532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7248.199404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7254.342433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7254.574097] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7263.190317] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7284.194721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7284.208657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7284.228973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7284.238253] Lustre: Skipped 1 previous similar message [ 7284.253222] Lustre: Skipped 1 previous similar message [ 7287.692898] Lustre: server umount lustre-OST0000 complete [ 7294.434467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.446524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7294.451015] Lustre: Skipped 1 previous similar message [ 7294.475147] Lustre: Skipped 1 previous similar message [ 7297.728513] Lustre: server umount lustre-MDT0000 complete [ 7299.554920] LustreError: 213918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7299.561924] LustreError: 213918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7301.752617] LustreError: 213900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444733 with bad export cookie 13496844927701701515 [ 7301.764500] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7302.013773] Lustre: server umount lustre-MDT0001 complete [ 7311.941151] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 7314.446350] Key type lgssc unregistered [ 7314.747942] LNet: 217601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7314.765984] LNetError: 217601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7315.813920] LNet: Removed LNI 192.168.202.125@tcp [ 7316.570829] Key type .llcrypt unregistered [ 7316.572693] Key type ._llcrypt unregistered [ 7339.619843] Key type ._llcrypt registered [ 7339.623720] Key type .llcrypt registered [ 7339.817792] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 7341.013717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7341.045547] alg: No test for adler32 (adler32-zlib) [ 7342.127561] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7342.374097] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 7344.055718] Key type lgssc registered [ 7346.138253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7357.129635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7357.152665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7358.877705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7362.895499] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7369.184018] LustreError: 218864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7369.210924] LustreError: 218864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7371.413900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.874928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7376.933765] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7385.921692] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7393.025489] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7401.104494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7401.615558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.687267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7409.484088] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7418.790642] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7428.403095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7428.460938] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7428.675517] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7435.197471] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7443.294135] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7445.226302] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7457.235315] Lustre: Permanently deactivating lustre-OST0001 [ 7457.260768] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7465.523034] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7476.361667] Lustre: Permanently reactivating lustre-OST0001 [ 7476.369912] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7476.372601] Lustre: Skipped 2 previous similar messages [ 7486.019986] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7486.036810] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7486.057993] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7486.070166] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7486.138209] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7491.045906] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7491.065063] Lustre: Skipped 1 previous similar message [ 7491.070176] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7491.075220] Lustre: Skipped 1 previous similar message [ 7492.930984] Lustre: server umount lustre-OST0001 complete [ 7496.175226] LustreError: 221308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7496.181383] LustreError: 221308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7499.238501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7499.253340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.289097] Lustre: Skipped 1 previous similar message [ 7499.306343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7501.280794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7501.284923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7501.295242] LustreError: 220684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7501.323254] Lustre: Skipped 1 previous similar message [ 7504.393540] Lustre: server umount lustre-OST0000 complete [ 7509.984399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7509.989992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.010192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7516.642969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7516.645853] Lustre: Skipped 2 previous similar messages [ 7524.319312] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7524.496247] Lustre: server umount lustre-MDT0000 complete [ 7525.868752] LustreError: 220103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.882939] LustreError: 220103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7529.439637] LustreError: 218844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444960 with bad export cookie 8518719616320008292 [ 7529.449548] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7529.459165] LustreError: 218844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7530.102711] Lustre: server umount lustre-MDT0001 complete [ 7542.671988] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 7545.428696] Key type lgssc unregistered [ 7545.764295] LNet: 223969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7545.777472] LNetError: 223969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7545.819872] LNet: Removed LNI 192.168.202.125@tcp [ 7547.141614] Key type .llcrypt unregistered [ 7547.144244] Key type ._llcrypt unregistered [ 7595.553600] Key type ._llcrypt registered [ 7595.557530] Key type .llcrypt registered [ 7595.711565] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 7597.457917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7597.489204] alg: No test for adler32 (adler32-zlib) [ 7598.710972] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7598.928146] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 7600.608170] Key type lgssc registered [ 7602.312189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7611.454935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7611.492126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7612.893231] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7612.911134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7613.107142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7618.986537] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7629.544225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.631184] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7629.670828] Lustre: 226170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7629.934948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7635.136807] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7643.632343] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7649.589429] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7656.329825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7656.510263] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7656.735388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7661.989763] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7663.232680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7670.675634] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7674.566941] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7679.379468] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7690.136137] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7699.007744] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7707.575882] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7724.406062] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7724.417597] Lustre: Skipped 1 previous similar message [ 7759.997857] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7760.005731] Lustre: Skipped 3 previous similar messages [ 7830.180765] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7830.192143] Lustre: Skipped 8 previous similar messages [ 7862.752677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7862.765269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7862.792182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7863.272676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.273684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7863.300450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7863.300460] Lustre: Skipped 1 previous similar message [ 7867.280906] Lustre: server umount lustre-OST0000 complete [ 7873.505483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7873.521120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7873.535878] Lustre: Skipped 1 previous similar message [ 7873.557821] Lustre: Skipped 1 previous similar message [ 7878.631712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7878.643667] Lustre: Skipped 1 previous similar message [ 7879.174836] Lustre: server umount lustre-MDT0000 complete [ 7883.436324] LustreError: 226162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445314 with bad export cookie 1056833935417975165 [ 7883.439893] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7883.450047] LustreError: 226162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7883.791662] Lustre: server umount lustre-MDT0001 complete [ 7895.601863] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 7898.468625] Key type lgssc unregistered [ 7898.777802] LNet: 230983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7898.787508] LNetError: 230983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7898.804634] LNet: Removed LNI 192.168.202.125@tcp [ 7899.811923] Key type .llcrypt unregistered [ 7899.813995] Key type ._llcrypt unregistered [ 7925.830891] Key type ._llcrypt registered [ 7925.835417] Key type .llcrypt registered [ 7925.961602] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 7927.564315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7927.603907] alg: No test for adler32 (adler32-zlib) [ 7929.058271] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7929.524715] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 7931.343426] Key type lgssc registered [ 7932.886996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7942.192966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7942.221844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7943.859903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7949.557855] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7954.402090] LustreError: 232395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7954.409018] LustreError: 232395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7958.554781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7958.971883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7963.333587] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 7971.876920] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7978.471792] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7986.631891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.104251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7989.191456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7993.705570] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_default_debug -1 all [ 8002.477355] Lustre: DEBUG MARKER: oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8030.179045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8030.180133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8030.192986] LustreError: Skipped 1 previous similar message [ 8030.195963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.260654] Lustre: Skipped 1 previous similar message [ 8032.660921] Lustre: server umount lustre-OST0000 complete [ 8040.416760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8040.422629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8040.437116] Lustre: Skipped 1 previous similar message [ 8040.438547] Lustre: Skipped 1 previous similar message [ 8045.537197] LustreError: 232394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8045.982373] Lustre: server umount lustre-MDT0000 complete [ 8050.664642] LustreError: 232395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.679153] LustreError: 232395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8051.301541] LustreError: 232376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445482 with bad export cookie 1007775778514581121 [ 8051.309133] LustreError: MGC192.168.202.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8051.318520] LustreError: 232376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8051.821457] Lustre: server umount lustre-MDT0001 complete [ 8064.679973] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 8070.185490] Key type lgssc unregistered [ 8070.543161] LNet: 236086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8070.547652] LNetError: 236086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8070.560515] LNet: Removed LNI 192.168.202.125@tcp [ 8071.488594] Key type .llcrypt unregistered [ 8071.499155] Key type ._llcrypt unregistered [ 8099.815876] Key type ._llcrypt registered [ 8099.819909] Key type .llcrypt registered [ 8099.970817] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 8101.909792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8101.990430] alg: No test for adler32 (adler32-zlib) [ 8103.365429] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8103.707109] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 8105.455278] Key type lgssc registered [ 8107.091340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8385.642878] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 8389.553991] Key type lgssc unregistered [ 8389.961694] LNet: 238108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8389.972854] LNetError: 238108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8390.004813] LNet: Removed LNI 192.168.202.125@tcp [ 8391.141517] Key type .llcrypt unregistered [ 8391.142539] Key type ._llcrypt unregistered [ 8417.445472] Key type ._llcrypt registered [ 8417.449739] Key type .llcrypt registered [ 8417.591613] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing set_hostid [ 8435.406819] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing load_modules_local [ 8436.495088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8436.601273] alg: No test for adler32 (adler32-zlib) [ 8437.594921] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8437.899837] LNet: Added LNI 192.168.202.125@tcp [8/256/0/180] [ 8439.663187] Key type lgssc registered [ 8440.904264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8476.838968] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8210 sec ========== 10:31:45 (1788445905) [ 8479.270530] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:31:47 (1788445907) === [ 8483.618786] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:31:52 (1788445912) === [ 8527.204287] Lustre: DEBUG MARKER: oleg225-server.virtnet: executing unload_modules_local [ 8529.985844] Key type lgssc unregistered [ 8530.284907] LNet: 243344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8530.294793] LNetError: 243344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8530.325469] LNet: Removed LNI 192.168.202.125@tcp [ 8531.217885] Key type .llcrypt unregistered [ 8531.224831] Key type ._llcrypt unregistered