[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509571010 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002385] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003000] kvm-guest: setup PV IPIs [ 0.003000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.003000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.004026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.005032] pid_max: default: 32768 minimum: 301 [ 0.007152] LSM: Security Framework initializing [ 0.009047] Yama: becoming mindful. [ 0.010049] SELinux: Initializing. [ 0.011093] *** VALIDATE selinux *** [ 0.027303] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032879] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033114] *** VALIDATE tmpfs *** [ 0.034498] *** VALIDATE proc *** [ 0.035252] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.038198] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041024] Spectre V2 : User space: Vulnerable [ 0.042006] Speculative Store Bypass: Vulnerable [ 0.045317] debug: unmapping init [mem 0xffffffffa2059000-0xffffffffa2060fff] [ 0.047148] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048762] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050011] ... bit width: 48 [ 0.051009] ... generic registers: 4 [ 0.052010] ... value mask: 0000ffffffffffff [ 0.053012] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055010] ... event mask: 000000070000000f [ 0.056250] rcu: Hierarchical SRCU implementation. [ 0.058524] smp: Bringing up secondary CPUs ... [ 0.059627] x86: Booting SMP configuration: [ 0.060020] .... node #0, CPUs: #1 #2 #3 [ 0.064065] smp: Brought up 1 node, 4 CPUs [ 0.066015] smpboot: Max logical packages: 1 [ 0.067009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.101063] node 0 deferred pages initialised in 30ms [ 0.104594] devtmpfs: initialized [ 0.106895] x86/mm: Memory block size: 128MB [ 0.108000] gcov: version magic: 0x41383552 [ 0.108000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.108000] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.108000] pinctrl core: initialized pinctrl subsystem [ 0.108000] [ 0.108000] ************************************************************* [ 0.108000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.108000] ** ** [ 0.108000] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.108000] ** ** [ 0.108000] ** This means that this kernel is built to expose internal ** [ 0.108000] ** IOMMU data structures, which may compromise security on ** [ 0.108000] ** your system. ** [ 0.112012] ** ** [ 0.113000] ** If you see this message and you are not debugging the ** [ 0.117012] ** kernel, report this immediately to your vendor! ** [ 0.121010] ** ** [ 0.122006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124007] ************************************************************* [ 0.126761] NET: Registered protocol family 16 [ 0.128361] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.130046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.133056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.137091] cpuidle: using governor menu [ 0.140613] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.143584] PCI: Using configuration type 1 for base access [ 0.146132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.157101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.158018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.161092] cryptd: max_cpu_qlen set to 1000 [ 0.164311] ACPI: Added _OSI(Module Device) [ 0.166012] ACPI: Added _OSI(Processor Device) [ 0.168010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.169000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.169000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.171654] ACPI: Interpreter enabled [ 0.172000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.172000] ACPI: Using IOAPIC for interrupt routing [ 0.172000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.172373] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.182708] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.185022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.187011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.190065] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.194441] acpiphp: Slot [2] registered [ 0.196102] acpiphp: Slot [5] registered [ 0.197104] acpiphp: Slot [6] registered [ 0.198171] acpiphp: Slot [7] registered [ 0.200081] acpiphp: Slot [8] registered [ 0.201105] acpiphp: Slot [9] registered [ 0.202091] acpiphp: Slot [10] registered [ 0.204085] acpiphp: Slot [3] registered [ 0.205071] acpiphp: Slot [4] registered [ 0.206059] acpiphp: Slot [11] registered [ 0.207087] acpiphp: Slot [12] registered [ 0.209101] acpiphp: Slot [13] registered [ 0.210070] acpiphp: Slot [14] registered [ 0.211112] acpiphp: Slot [15] registered [ 0.213082] acpiphp: Slot [16] registered [ 0.214060] acpiphp: Slot [17] registered [ 0.216063] acpiphp: Slot [18] registered [ 0.217083] acpiphp: Slot [19] registered [ 0.218101] acpiphp: Slot [20] registered [ 0.220073] acpiphp: Slot [21] registered [ 0.221166] acpiphp: Slot [22] registered [ 0.223066] acpiphp: Slot [23] registered [ 0.224181] acpiphp: Slot [24] registered [ 0.226062] acpiphp: Slot [25] registered [ 0.228095] acpiphp: Slot [26] registered [ 0.230107] acpiphp: Slot [27] registered [ 0.231000] acpiphp: Slot [28] registered [ 0.231000] acpiphp: Slot [29] registered [ 0.232091] acpiphp: Slot [30] registered [ 0.234076] acpiphp: Slot [31] registered [ 0.235070] PCI host bridge to bus 0000:00 [ 0.236014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.239018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.241012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.243015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.244013] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.246017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.248155] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.249950] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.251144] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.258013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.261805] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.264014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.266012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.268012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.269564] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.272704] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.274033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.276759] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.279890] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.289016] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.292014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.297692] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.312010] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.326051] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.359029] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.378152] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.391022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.398974] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.423032] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.434900] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.444035] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.453019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.474026] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.489618] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.498034] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.505018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.521024] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.531562] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.538027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.545024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.558041] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.570081] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.576025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.583021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.598047] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.608000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.611443] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.612415] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.614459] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.617291] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.623161] iommu: Default domain type: Passthrough [ 0.626027] SCSI subsystem initialized [ 0.628390] ACPI: bus type USB registered [ 0.630173] usbcore: registered new interface driver usbfs [ 0.634178] usbcore: registered new interface driver hub [ 0.638165] usbcore: registered new device driver usb [ 0.640300] pps_core: LinuxPPS API ver. 1 registered [ 0.642017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.645110] PTP clock support registered [ 0.648244] EDAC MC: Ver: 3.0.0 [ 0.651000] PCI: Using ACPI for IRQ routing [ 0.653398] NetLabel: Initializing [ 0.656021] NetLabel: domain hash size = 128 [ 0.658013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.661182] NetLabel: unlabeled traffic allowed by default [ 0.664184] vgaarb: loaded [ 0.667713] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.671021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.676790] clocksource: Switched to clocksource kvm-clock [ 0.795284] VFS: Disk quotas dquot_6.6.0 [ 0.797098] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.799722] *** VALIDATE ramfs *** [ 0.801192] *** VALIDATE hugetlbfs *** [ 0.802625] pnp: PnP ACPI init [ 0.805180] pnp: PnP ACPI: found 6 devices [ 0.840596] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.844248] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.846581] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.848797] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.852110] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.854621] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.857739] NET: Registered protocol family 2 [ 0.860176] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.865337] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.869424] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.874967] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.878352] TCP: Hash tables configured (established 65536 bind 65536) [ 0.881187] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.884076] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.886804] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.889917] NET: Registered protocol family 1 [ 0.894788] RPC: Registered named UNIX socket transport module. [ 0.897387] RPC: Registered udp transport module. [ 0.899032] RPC: Registered tcp transport module. [ 0.900703] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.903425] NET: Registered protocol family 44 [ 0.905202] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.907211] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.909377] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.911602] PCI: CLS 0 bytes, default 64 [ 0.913326] Unpacking initramfs... [ 2.418081] hrtimer: interrupt took 8390581 ns [ 3.018873] debug: unmapping init [mem 0xffff8bd0bcc54000-0xffff8bd0bffbffff] [ 3.024389] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.025942] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.028994] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.621277] Initialise system trusted keyrings [ 3.623021] Key type blacklist registered [ 3.626876] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.637318] zbud: loaded [ 3.641108] *** VALIDATE nfs *** [ 3.642320] *** VALIDATE nfs4 *** [ 3.643913] pstore: using deflate compression [ 3.647426] Platform Keyring initialized [ 3.790722] NET: Registered protocol family 38 [ 3.792747] Key type asymmetric registered [ 3.795195] Asymmetric key parser 'x509' registered [ 3.796825] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.800672] io scheduler mq-deadline registered [ 3.803605] io scheduler kyber registered [ 3.806661] io scheduler bfq registered [ 3.808314] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.811404] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.814674] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.818082] ACPI: Power Button [PWRF] [ 3.917613] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.055573] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.369957] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.506896] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.733810] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.857847] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.902261] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.910908] Non-volatile memory driver v1.3 [ 4.912943] Linux agpgart interface v0.103 [ 4.946694] virtio_blk virtio1: [vda] 67976 512-byte logical blocks (34.8 MB/33.2 MiB) [ 4.950709] vda: detected capacity change from 0 to 34803712 [ 5.660857] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.663938] vdb: detected capacity change from 0 to 1073741824 [ 5.741526] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.744265] vdc: detected capacity change from 0 to 2621440000 [ 5.856854] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.864226] vdd: detected capacity change from 0 to 2621440000 [ 5.903599] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.909828] vde: detected capacity change from 0 to 4294967296 [ 5.924420] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.927343] vdf: detected capacity change from 0 to 4294967296 [ 5.957833] libphy: Fixed MDIO Bus: probed [ 5.962873] usbcore: registered new interface driver usbserial_generic [ 5.968355] usbserial: USB Serial support registered for generic [ 5.970607] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.974492] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.976069] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.978726] mousedev: PS/2 mouse device common for all mice [ 5.983774] rtc_cmos 00:05: RTC can wake from S4 [ 5.994281] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.998245] rtc_cmos 00:05: registered as rtc0 [ 5.998320] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.998356] intel_pstate: CPU model not supported [ 6.001801] hid: raw HID events driver (C) Jiri Kosina [ 6.014855] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.021045] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.025393] usbcore: registered new interface driver usbhid [ 6.027428] usbhid: USB HID core driver [ 6.028911] drop_monitor: Initializing network drop monitor service [ 6.031476] Initializing XFRM netlink socket [ 6.033393] NET: Registered protocol family 10 [ 6.038088] Segment Routing with IPv6 [ 6.039499] NET: Registered protocol family 17 [ 6.042427] mpls_gso: MPLS GSO support [ 6.047089] RAS: Correctable Errors collector initialized. [ 6.049150] AVX version of gcm_enc/dec engaged. [ 6.050782] AES CTR mode by8 optimization enabled [ 6.153178] sched_clock: Marking stable (6153150221, 0)->(7294842764, -1141692543) [ 6.159240] registered taskstats version 1 [ 6.163823] Loading compiled-in X.509 certificates [ 6.167158] zswap: loaded using pool lzo/zbud [ 6.197471] Key type big_key registered [ 6.211757] Key type encrypted registered [ 6.213174] ima: No TPM chip found, activating TPM-bypass! [ 6.215489] ima: Allocated hash algorithm: sha1 [ 6.218169] ima: No architecture policies found [ 6.220035] evm: Initialising EVM extended attributes: [ 6.221776] evm: security.selinux [ 6.223879] evm: security.ima [ 6.225697] evm: security.capability [ 6.227270] evm: HMAC attrs: 0x1 [ 6.230402] rtc_cmos 00:05: setting system clock to 2025-10-22 08:20:21 UTC (1761121221) [ 6.237524] debug: unmapping init [mem 0xffffffffa3003000-0xffffffffa31fffff] [ 6.240613] debug: unmapping init [mem 0xffffffffa1d82000-0xffffffffa2058fff] [ 6.247703] Write protecting the kernel read-only data: 28672k [ 6.251901] debug: unmapping init [mem 0xffffffffa0403000-0xffffffffa05fffff] [ 6.255160] debug: unmapping init [mem 0xffffffffa0d14000-0xffffffffa0dfffff] [ 6.296835] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.305370] systemd[1]: Detected virtualization kvm. [ 6.314637] systemd[1]: Detected architecture x86-64. [ 6.316482] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.347905] systemd[1]: No hostname configured. [ 6.349664] systemd[1]: Set hostname to . [ 6.351633] random: systemd: uninitialized urandom read (16 bytes read) [ 6.354135] systemd[1]: Initializing machine ID from random generator. [ 6.460057] random: ln: uninitialized urandom read (6 bytes read) [ 6.584857] random: systemd: uninitialized urandom read (16 bytes read) [ 6.589610] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.592681] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.603142] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.901445] device-mapper: uevent: version 1.0.3 [ 7.903540] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.641677] virtio_net virtio0 ens2: renamed from eth0 [ 9.758096] random: fast init done [ 10.787676] scsi host0: ata_piix [ 10.894620] scsi host1: ata_piix [ 10.896887] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.905775] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.746266] random: crng init done [ 14.747774] random: 7 urandom warning(s) missed due to ratelimiting [ 16.237433] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 18.399726] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.627113] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.969430] SELinux: Disabled at runtime. [ 21.030210] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.039090] systemd[1]: Detected virtualization kvm. [ 21.042878] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.112424] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.117202] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.126495] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.131340] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.135095] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.155216] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.184388] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ 22.729676] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 23.414538] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.009404] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.064563] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.514881] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.583700] EDAC sbridge: Ver: 1.1.2 [ 28.402988] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 29.128626] NFS: Registering the id_resolver key type [ 29.132877] Key type id_resolver registered [ 29.138774] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg126-server login: [ 55.126792] libcfs: loading out-of-tree module taints kernel. [ 55.141318] alg: No test for adler32 (adler32-zlib) [ 55.893377] Key type ._llcrypt registered [ 55.894946] Key type .llcrypt registered [ 55.930798] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 62.796331] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 63.320549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.582745] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 63.906147] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 63.908378] LNet: Accept secure, port 988 [ 65.535122] Key type lgssc registered [ 65.992052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.470704] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 73.873767] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.981346] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.845887] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.889470] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.720281] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 94.394768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.437561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 95.530480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 95.543671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 95.582379] Lustre: lustre-MDT0000: new disk, initializing [ 95.605833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 95.613073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 97.033049] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 102.208024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.233755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 102.264480] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 102.272444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 102.275161] Lustre: Skipped 1 previous similar message [ 102.312873] Lustre: lustre-MDT0001: new disk, initializing [ 102.335085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 102.344193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 102.348742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 103.751877] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 108.728201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.752376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 108.755094] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 108.844162] Lustre: lustre-OST0000: new disk, initializing [ 108.845719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 108.866515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 110.218679] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 110.827732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 110.831118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 115.194854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.217027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 115.221234] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 115.256418] Lustre: lustre-OST0001: new disk, initializing [ 115.257843] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 115.273236] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 116.672450] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 116.841233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 116.843895] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 122.085417] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 126.619363] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 133.000561] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing check_logdir /tmp/testlogs/ [ 134.611919] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing yml_node [ 136.088258] Lustre: DEBUG MARKER: Client: 2.15.7.9 [ 136.956125] Lustre: DEBUG MARKER: MDS: 2.15.7.9 [ 137.766601] Lustre: DEBUG MARKER: OSS: 2.15.7.9 [ 138.287026] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Oct 22 04:22:33 EDT 2025 [ 141.121219] Lustre: DEBUG MARKER: excepting tests: 32newtarball 110 [ 141.641146] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 153.567450] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 153.569337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 153.572109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 158.175723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 158.175834] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 158.178693] Lustre: Skipped 2 previous similar messages [ 158.185174] Lustre: Skipped 2 previous similar messages [ 159.297134] Lustre: server umount lustre-MDT0000 complete [ 160.534637] LustreError: 7228:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121375 with bad export cookie 17897098744566393304 [ 160.535746] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 160.538191] LustreError: 7228:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 163.295538] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.295729] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 163.295744] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 163.298413] Lustre: Skipped 1 previous similar message [ 163.299764] Lustre: Skipped 1 previous similar message [ 165.343717] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 165.343892] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 165.346310] Lustre: Skipped 2 previous similar messages [ 165.351731] LustreError: Skipped 2 previous similar messages [ 166.367876] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 166.370792] LustreError: Skipped 1 previous similar message [ 166.699725] Lustre: server umount lustre-MDT0001 complete [ 174.120717] Lustre: server umount lustre-OST0000 complete [ 181.601834] Lustre: server umount lustre-OST0001 complete [ 184.204779] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 187.577767] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 191.389370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 193.712893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.049433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 198.289882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.792952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.819685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.911919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 200.925529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 200.965141] Lustre: lustre-MDT0000: new disk, initializing [ 200.996530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.003980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 202.680478] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 207.346930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 207.372795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.415518] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 207.542992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.292584] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 213.945327] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 216.349062] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 219.322547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 219.357781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.361310] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 219.470365] Lustre: lustre-OST0000: new disk, initializing [ 219.471912] Lustre: Skipped 1 previous similar message [ 219.474412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 219.477109] Lustre: Skipped 2 previous similar messages [ 219.508647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.512297] Lustre: Skipped 1 previous similar message [ 221.054048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 221.057246] Lustre: Skipped 1 previous similar message [ 221.059670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 221.368688] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 225.654130] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 227.506891] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 227.615812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 229.534392] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 229.629709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 231.393048] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 231.393352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 231.396381] LustreError: Skipped 1 previous similar message [ 231.403075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 231.406718] Lustre: Skipped 3 previous similar messages [ 236.544556] Lustre: server umount lustre-OST0000 complete [ 238.560197] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 238.564564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 238.570950] Lustre: Skipped 1 previous similar message [ 241.631912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 241.635164] Lustre: Skipped 2 previous similar messages [ 252.383120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 252.432743] Lustre: server umount lustre-MDT0000 complete [ 253.766927] LustreError: 15156:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121469 with bad export cookie 17897098744566394746 [ 253.769469] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 253.769730] LustreError: 15156:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 255.840086] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:24:30 (1761121470) [ 262.890874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 263.078209] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 263.105803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.727519] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 268.181916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 268.256184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 268.259686] Lustre: Skipped 4 previous similar messages [ 270.274044] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 273.371621] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 275.731439] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 276.624252] Lustre: Failing over lustre-MDT0000 [ 276.712930] Lustre: server umount lustre-MDT0000 complete [ 276.715751] Lustre: Skipped 1 previous similar message [ 278.496626] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 278.496723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 278.497318] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 278.507697] Lustre: Skipped 2 previous similar messages [ 285.663119] Lustre: 3334:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761121493/real 1761121493] req@00000000e5ed3b38 x1846669506305792/t0(0) o400->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761121500 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 285.673272] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 290.784741] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.790740] LustreError: Skipped 6 previous similar messages [ 290.809092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 302.050271] Lustre: Evicted from MGS (at 192.168.201.126@tcp) after server handle changed from 0xf85f447c3bf832d0 to 0xf85f447c3bf834eb [ 302.055812] Lustre: MGC192.168.201.126@tcp: Connection restored to (at 0@lo) [ 302.159110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 302.162384] Lustre: Skipped 1 previous similar message [ 304.032150] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 307.168793] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 307.170976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 307.177760] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 308.603762] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 312.287692] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 312.293580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.301899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.087201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 325.134281] Lustre: server umount lustre-MDT0000 complete [ 326.763147] LustreError: 20106:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121542 with bad export cookie 17897098744566396139 [ 326.764992] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 326.767573] LustreError: 20106:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 330.726318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 330.925164] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.934228] LustreError: Skipped 6 previous similar messages [ 332.844992] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 336.682765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.839174] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 342.063920] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 344.622668] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 345.622203] Lustre: Failing over lustre-MDT0000 [ 347.104172] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 347.104343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.116254] Lustre: Skipped 2 previous similar messages [ 354.271178] Lustre: 3332:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761121562/real 1761121562] req@00000000315aa2d0 x1846669506318272/t0(0) o400->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761121569 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 354.283220] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.568987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 365.536996] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.545120] LustreError: Skipped 12 previous similar messages [ 371.682756] Lustre: Evicted from MGS (at 192.168.201.126@tcp) after server handle changed from 0xf85f447c3bf8367a to 0xf85f447c3bf8388e [ 371.692894] Lustre: MGC192.168.201.126@tcp: Connection restored to (at 0@lo) [ 371.695468] Lustre: Skipped 1 previous similar message [ 371.861181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.864415] Lustre: Skipped 2 previous similar messages [ 373.868199] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 376.996873] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 377.002500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 377.011422] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 379.210826] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 382.435557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 382.438556] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 382.440258] Lustre: Skipped 7 previous similar messages [ 396.255321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 396.328819] Lustre: server umount lustre-MDT0000 complete [ 396.332144] Lustre: Skipped 2 previous similar messages [ 398.859252] LustreError: 22850:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121614 with bad export cookie 17897098744566397070 [ 398.860293] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 398.865690] LustreError: 22850:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 404.145299] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:26:58 (1761121618) [ 412.323722] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 416.728349] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 421.683068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 424.977829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 428.274046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 431.621643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 436.229646] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 441.780241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 441.829229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.006768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 442.048296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 442.110816] Lustre: lustre-MDT0000: new disk, initializing [ 442.208136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 444.510067] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 449.710709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 449.739395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.922287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 451.909799] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 456.728194] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 459.404780] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 462.218845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 462.249752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.253824] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 463.632931] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 464.533420] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 469.606893] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 473.216481] Lustre: lustre-OST0000-osc-MDT0001:0x280000400 is used up. Update to new seq [ 473.220265] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000401 [ 479.201174] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 479.203748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.211165] Lustre: Skipped 3 previous similar messages [ 494.560196] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.565401] LustreError: Skipped 7 previous similar messages [ 494.838991] LustreError: 29577:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761121710 with bad export cookie 17897098744566397476 [ 494.843329] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 494.845956] LustreError: 29577:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 500.874138] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 502.146786] Key type lgssc unregistered [ 502.299336] LNet: 33636:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.304862] LNet: Removed LNI 192.168.201.126@tcp [ 502.739361] Key type .llcrypt unregistered [ 502.741976] Key type ._llcrypt unregistered [ 514.122575] alg: No test for adler32 (adler32-zlib) [ 514.874549] Key type ._llcrypt registered [ 514.876236] Key type .llcrypt registered [ 514.933545] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 521.449595] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 521.855853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 522.162383] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 522.259959] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 522.262607] LNet: Accept secure, port 988 [ 523.855124] Key type lgssc registered [ 524.323817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 528.341560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 531.353219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 534.128603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 537.094473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 541.415141] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 546.260440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 546.294089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.406853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 547.423197] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 547.468837] Lustre: lustre-MDT0000: new disk, initializing [ 547.503332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.513298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 549.395349] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 554.351151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 554.383244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.433633] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 554.462153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 554.465978] Lustre: Skipped 1 previous similar message [ 554.511516] Lustre: lustre-MDT0001: new disk, initializing [ 554.558398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 554.577087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 554.583266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 556.418929] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 560.923746] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 563.209739] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 565.840962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 565.868863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 565.872374] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 565.974313] Lustre: lustre-OST0000: new disk, initializing [ 565.976400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 566.007890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 566.973745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 566.977654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 566.994659] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 567.722500] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 573.046900] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 576.020799] Lustre: DEBUG MARKER: On OST0, 49667 inodes available. Want 100500. rc=0 [ 577.163219] Lustre: 39858:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 577.169433] Lustre: 39858:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 577.172824] Lustre: 39858:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 577.180237] Lustre: 39858:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 577.185928] Lustre: 39858:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 577.194908] Lustre: 39858:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 577.803277] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 1105, rollback = 9 [ 577.808057] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 159 previous similar messages [ 577.813617] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 65/260/0, destroy: 0/0/0 [ 577.818403] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 159 previous similar messages [ 577.824161] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 577.828440] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 159 previous similar messages [ 577.832580] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 577.837448] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 159 previous similar messages [ 577.841914] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 65/1105/0, delete: 0/0/0 [ 577.847302] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 159 previous similar messages [ 577.852751] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 577.855852] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 159 previous similar messages [ 579.230914] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 2176, rollback = 9 [ 579.236186] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 320 previous similar messages [ 579.240683] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 128/512/1, destroy: 0/0/0 [ 579.245220] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 320 previous similar messages [ 579.248863] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 579.252805] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 320 previous similar messages [ 579.256344] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 579.260759] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 320 previous similar messages [ 579.265391] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 579.270330] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 320 previous similar messages [ 579.274467] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 579.278938] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 320 previous similar messages [ 581.413396] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 2176, rollback = 9 [ 581.419451] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 636 previous similar messages [ 581.425025] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 128/512/1, destroy: 0/0/0 [ 581.428547] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 636 previous similar messages [ 581.434287] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 581.439026] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 636 previous similar messages [ 581.443901] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 581.448088] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 636 previous similar messages [ 581.452719] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 581.458121] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 636 previous similar messages [ 581.463756] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 581.469484] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 636 previous similar messages [ 585.774564] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 2176, rollback = 9 [ 585.780957] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 1023 previous similar messages [ 585.785972] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 128/512/1, destroy: 0/0/0 [ 585.794369] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 1023 previous similar messages [ 585.798569] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 585.804914] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 1023 previous similar messages [ 585.810513] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 585.813693] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 1023 previous similar messages [ 585.819014] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 585.824301] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 1023 previous similar messages [ 585.829889] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 585.834954] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 1023 previous similar messages [ 594.241165] Lustre: 39858:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 2176, rollback = 9 [ 594.245604] Lustre: 39858:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 2041 previous similar messages [ 594.250450] Lustre: 39858:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 128/512/1, destroy: 0/0/0 [ 594.255659] Lustre: 39858:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 2041 previous similar messages [ 594.260081] Lustre: 39858:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 594.264429] Lustre: 39858:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 2041 previous similar messages [ 594.268376] Lustre: 39858:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 594.273189] Lustre: 39858:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 2041 previous similar messages [ 594.277055] Lustre: 39858:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 594.281797] Lustre: 39858:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 2041 previous similar messages [ 594.287868] Lustre: 39858:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 594.293910] Lustre: 39858:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 2041 previous similar messages [ 610.346438] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 2176, rollback = 9 [ 610.350871] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 3989 previous similar messages [ 610.354108] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 128/512/1, destroy: 0/0/0 [ 610.357468] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 3989 previous similar messages [ 610.362009] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 610.366531] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 3989 previous similar messages [ 610.371753] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 610.376336] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 3989 previous similar messages [ 610.382809] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 610.388333] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 3989 previous similar messages [ 610.393688] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 610.397629] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 3989 previous similar messages [ 651.030496] Lustre: 40819:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 2176, rollback = 9 [ 651.039275] Lustre: 40819:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 3033 previous similar messages [ 651.044307] Lustre: 40819:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 128/512/1, destroy: 0/0/0 [ 651.049808] Lustre: 40819:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 3033 previous similar messages [ 651.056589] Lustre: 40819:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 651.061772] Lustre: 40819:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 3033 previous similar messages [ 651.066458] Lustre: 40819:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 651.073334] Lustre: 40819:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 3033 previous similar messages [ 651.078478] Lustre: 40819:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 651.082407] Lustre: 40819:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 3033 previous similar messages [ 651.087458] Lustre: 40819:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 651.091462] Lustre: 40819:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 3033 previous similar messages [ 716.269677] Lustre: 40823:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 2176, rollback = 9 [ 716.275934] Lustre: 40823:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 10836 previous similar messages [ 716.281479] Lustre: 40823:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 128/512/1, destroy: 0/0/0 [ 716.287893] Lustre: 40823:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 10836 previous similar messages [ 716.292632] Lustre: 40823:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 716.295957] Lustre: 40823:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 10836 previous similar messages [ 716.301041] Lustre: 40823:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 716.305350] Lustre: 40823:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 10836 previous similar messages [ 716.309856] Lustre: 40823:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 716.314580] Lustre: 40823:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 10836 previous similar messages [ 716.317837] Lustre: 40823:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.321346] Lustre: 40823:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 10836 previous similar messages [ 845.215211] Lustre: 39709:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 2176, rollback = 9 [ 845.234371] Lustre: 39709:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 20737 previous similar messages [ 845.249675] Lustre: 39709:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 128/512/6, destroy: 0/0/0 [ 845.268273] Lustre: 39709:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 20737 previous similar messages [ 845.288830] Lustre: 39709:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 845.306729] Lustre: 39709:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 20737 previous similar messages [ 845.323777] Lustre: 39709:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 845.348392] Lustre: 39709:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 20737 previous similar messages [ 845.362632] Lustre: 39709:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 845.375177] Lustre: 39709:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 20737 previous similar messages [ 845.389404] Lustre: 39709:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 845.413900] Lustre: 39709:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 20737 previous similar messages [ 1119.251358] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 2176, rollback = 9 [ 1119.265502] Lustre: 39708:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 15357 previous similar messages [ 1119.281446] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 128/512/1, destroy: 0/0/0 [ 1119.284871] Lustre: 39708:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 15357 previous similar messages [ 1119.290709] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1119.304929] Lustre: 39708:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 15357 previous similar messages [ 1119.323458] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1119.343810] Lustre: 39708:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 15357 previous similar messages [ 1119.360025] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 1119.381893] Lustre: 39708:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 15357 previous similar messages [ 1119.403315] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1119.419046] Lustre: 39708:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 15357 previous similar messages [ 1632.489151] Lustre: 39709:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 2176, rollback = 9 [ 1632.499525] Lustre: 39709:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 27479 previous similar messages [ 1632.507393] Lustre: 39709:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 128/512/6, destroy: 0/0/0 [ 1632.516414] Lustre: 39709:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 27479 previous similar messages [ 1632.521231] Lustre: 39709:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1632.526861] Lustre: 39709:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 27479 previous similar messages [ 1632.532081] Lustre: 39709:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1632.538789] Lustre: 39709:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 27479 previous similar messages [ 1632.544753] Lustre: 39709:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 1632.552029] Lustre: 39709:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 27479 previous similar messages [ 1632.557055] Lustre: 39709:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1632.564806] Lustre: 39709:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 27479 previous similar messages [ 2233.572092] Lustre: 40819:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 516 < left 2176, rollback = 9 [ 2233.580695] Lustre: 40819:0:(osd_internal.h:1344:osd_trans_exec_op()) Skipped 39316 previous similar messages [ 2233.587334] Lustre: 40819:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 128/512/0, destroy: 0/0/0 [ 2233.591626] Lustre: 40819:0:(osd_handler.c:1944:osd_trans_dump_creds()) Skipped 39316 previous similar messages [ 2233.599063] Lustre: 40819:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 2233.603692] Lustre: 40819:0:(osd_handler.c:1951:osd_trans_dump_creds()) Skipped 39316 previous similar messages [ 2233.608888] Lustre: 40819:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/0, punch: 0/0/0, quota 1/3/0 [ 2233.615364] Lustre: 40819:0:(osd_handler.c:1961:osd_trans_dump_creds()) Skipped 39316 previous similar messages [ 2233.620326] Lustre: 40819:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 128/2176/0, delete: 0/0/0 [ 2233.623824] Lustre: 40819:0:(osd_handler.c:1968:osd_trans_dump_creds()) Skipped 39316 previous similar messages [ 2233.629949] Lustre: 40819:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2233.636260] Lustre: 40819:0:(osd_handler.c:1975:osd_trans_dump_creds()) Skipped 39316 previous similar messages [ 2386.422592] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 2388.142649] Lustre: Failing over lustre-OST0000 [ 2388.286471] Lustre: server umount lustre-OST0000 complete [ 2397.151315] Lustre: 35330:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761123605/real 1761123605] req@0000000004476fa1 x1846669993870592/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761123612 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2397.153325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.163458] Lustre: 35330:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2407.353671] LDISKFS-fs (dm-2): recovery complete [ 2407.358285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2407.620121] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 2407.640808] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2410.262829] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 2412.498688] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2414.838898] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 2414.839825] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 2414.840731] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 2414.861481] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4295077042 to 0x0:4295077121 [ 2418.401294] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2419.317127] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2430.433114] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.437796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.440578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.445749] Lustre: Skipped 2 previous similar messages [ 2434.183188] Lustre: server umount lustre-OST0000 complete [ 2434.527416] Lustre: 35329:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761123605/real 1761123605] req@000000006fd93b6f x1846669993870528/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761123649 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2438.625443] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2438.631888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.645454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2438.651975] Lustre: Skipped 1 previous similar message [ 2439.327163] Lustre: 35330:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761123610/real 1761123610] req@000000008796ea44 x1846669993871104/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761123654 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2440.672435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2440.673554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.695188] Lustre: 35329:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761123612/real 1761123612] req@00000000b2af4191 x1846669993871552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761123656 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2442.532607] Lustre: server umount lustre-MDT0000 complete [ 2444.927288] LustreError: 37633:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761123660 with bad export cookie 1005466967682220815 [ 2444.930917] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2444.933494] LustreError: 37633:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2444.942119] LustreError: 39857:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 2445.147147] Lustre: server umount lustre-MDT0001 complete [ 2452.427232] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2453.996284] Key type lgssc unregistered [ 2454.147766] LNet: 47689:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.151522] LNet: Removed LNI 192.168.201.126@tcp [ 2454.603467] Key type .llcrypt unregistered [ 2454.606954] Key type ._llcrypt unregistered [ 2465.658112] alg: No test for adler32 (adler32-zlib) [ 2466.412598] Key type ._llcrypt registered [ 2466.414151] Key type .llcrypt registered [ 2466.493574] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2466.951516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2467.335817] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 2467.482118] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2467.487403] LNet: Accept secure, port 988 [ 2469.103128] Key type lgssc registered [ 2469.798717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2474.339429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.567738] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.598879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.904633] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 2480.608587] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.096130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2483.362098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2485.953826] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 2490.120472] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2493.355158] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2497.275108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2497.563355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.662809] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 2503.603885] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2504.745783] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4295077222 to 0x0:4295077249 [ 2516.445947] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 05:02:11 (1761123731) [ 2517.595421] Lustre: Modifying parameter general.jobid_name in log params [ 2518.636958] Lustre: Modifying parameter general.jobid_name in log params [ 2522.814229] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 05:02:17 (1761123737) [ 2527.880984] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 05:02:22 (1761123742) [ 2529.037423] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb in log lustre-client [ 2534.264681] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 05:02:28 (1761123748) [ 2535.294645] Lustre: Setting parameter general.osc.*.max_pages_per_rpc in log params [ 2535.299053] Lustre: Skipped 2 previous similar messages [ 2540.242349] Lustre: Modifying parameter lustre-OST0000-osc.osc.max_dirty_mb in log lustre-client [ 2540.246750] Lustre: Skipped 4 previous similar messages [ 2547.986523] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 05:02:42 (1761123762) [ 2549.552771] Lustre: *** cfs_fail_loc=131b, val=0*** [ 2551.429640] Lustre: *** cfs_fail_loc=131b, val=0*** [ 2556.062837] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 05:02:50 (1761123770) [ 2557.188552] Lustre: Modifying parameter general.jobid_name in log params [ 2557.191449] Lustre: Skipped 6 previous similar messages [ 2568.963619] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:03:02 (1761123782) [ 2573.345531] Lustre: Modifying parameter general.timeout in log params [ 2573.357705] Lustre: Skipped 5 previous similar messages [ 2606.050628] Lustre: Modifying parameter general.timeout in log params [ 2606.062650] Lustre: Skipped 21 previous similar messages [ 2670.461171] Lustre: Modifying parameter general.timeout in log params [ 2670.464351] Lustre: Skipped 60 previous similar messages [ 2713.569873] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:05:27 (1761123927) [ 2729.441299] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2729.452242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.484092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.960060] Lustre: server umount lustre-MDT0000 complete [ 2733.537663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.543227] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.564198] Lustre: Skipped 1 previous similar message [ 2733.596781] LustreError: Skipped 2 previous similar messages [ 2734.361330] LustreError: 48982:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761123949 with bad export cookie 4319760253311353597 [ 2734.369370] LustreError: 48982:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2734.372253] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2734.626968] Lustre: server umount lustre-MDT0001 complete [ 2743.781603] Lustre: 63218:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761123953/real 1761123953] req@00000000fc3e4479 x1846672026059072/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761123959 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'jobid_name=jobid_name=umount.ro' [ 2743.953958] Lustre: server umount lustre-OST0000 complete [ 2751.707847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2754.836428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2758.704849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2762.476208] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2769.101455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.419365] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2769.668230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2774.187396] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 2782.542353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.659317] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2783.075902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2787.874698] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 2795.538352] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2801.169263] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2807.617375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2807.858507] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2808.245301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2813.574171] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 2816.143516] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4295077222 to 0x0:4295077281 [ 2822.128777] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2823.966453] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall in log params [ 2823.970563] Lustre: Skipped 48 previous similar messages [ 2831.329087] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2831.339465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.364747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2836.466795] Lustre: server umount lustre-OST0000 complete [ 2842.084336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2842.090044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2842.095681] Lustre: Skipped 2 previous similar messages [ 2842.104075] Lustre: Skipped 1 previous similar message [ 2846.564142] Lustre: server umount lustre-MDT0000 complete [ 2847.200784] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.212824] LustreError: Skipped 1 previous similar message [ 2850.513261] LustreError: 64659:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124065 with bad export cookie 4319760253311356992 [ 2850.519618] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2850.618978] LustreError: 64659:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2863.993651] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2867.231463] Key type lgssc unregistered [ 2867.626875] LNet: 68589:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2867.643445] LNet: Removed LNI 192.168.201.126@tcp [ 2868.657398] Key type .llcrypt unregistered [ 2868.661374] Key type ._llcrypt unregistered [ 2936.970355] alg: No test for adler32 (adler32-zlib) [ 2937.736898] Key type ._llcrypt registered [ 2937.747475] Key type .llcrypt registered [ 2937.933870] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 2938.290877] Key type .llcrypt unregistered [ 2938.302133] Key type ._llcrypt unregistered [ 2943.096028] alg: No test for adler32 (adler32-zlib) [ 2943.850480] Key type ._llcrypt registered [ 2943.857613] Key type .llcrypt registered [ 2943.991480] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2949.300378] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules [ 2951.079346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2951.725728] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 2952.079375] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 2952.088281] LNet: Accept secure, port 988 [ 2953.887102] Key type lgssc registered [ 2955.378455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2956.773274] LustreError: 72953:0:(libcfs_fail.h:202:cfs_race_wait()) cfs_race id 60d sleeping [ 2973.574945] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 2978.113266] LustreError: 72953:0:(libcfs_fail.h:205:cfs_race_wait()) cfs_fail_race id 60d awake: rc=0 [ 2985.844565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2987.351805] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.456488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2992.586739] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 2992.609171] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.732328] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3001.831000] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.285878] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:10:16 (1761124216) [ 3006.951125] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.389575] LustreError: 74040:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3007.402200] LustreError: 74040:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3007.409983] LustreError: 74040:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 3013.599147] Lustre: 74883:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761124222/real 1761124222] req@00000000cc17192a x1846672534540224/t0(0) o251->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761124228 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3013.799063] Lustre: server umount lustre-MDT0000 complete [ 3028.536313] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3031.931442] Key type lgssc unregistered [ 3032.286407] LNet: 75639:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3032.297071] LNet: Removed LNI 192.168.201.126@tcp [ 3033.528738] Key type .llcrypt unregistered [ 3033.532524] Key type ._llcrypt unregistered [ 3051.226257] alg: No test for adler32 (adler32-zlib) [ 3051.989509] Key type ._llcrypt registered [ 3051.993855] Key type .llcrypt registered [ 3052.187966] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3053.294271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3053.806640] Lustre: Lustre: Build Version: 2.15.7_9_g826bc77 [ 3054.105593] LNet: Added LNI 192.168.201.126@tcp [8/256/0/180] [ 3054.110258] LNet: Accept secure, port 988 [ 3055.825339] Key type lgssc registered [ 3056.857179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3064.607470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.017354] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.122649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.276389] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3071.457423] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3076.580517] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.017600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.651032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3084.141697] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3091.521327] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3097.916510] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3105.217533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3105.708855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.116675] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3115.967475] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4295077222 to 0x0:4295077313 [ 3118.040337] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3166.870889] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:13:01 (1761124381) [ 3168.270338] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 3170.077911] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:13:04 (1761124384) [ 3176.937714] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3176.947577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.987389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3179.966632] Lustre: server umount lustre-MDT0000 complete [ 3182.561518] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3182.581609] LustreError: Skipped 2 previous similar messages [ 3183.126067] LustreError: 76914:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124398 with bad export cookie 7120615748966568297 [ 3183.129933] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3183.136587] LustreError: 76914:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3183.414077] Lustre: server umount lustre-MDT0001 complete [ 3193.312582] Lustre: 80425:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761124402/real 1761124402] req@00000000cab1a9d4 x1846672641515968/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761124408 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3193.472197] Lustre: server umount lustre-OST0000 complete [ 3202.978596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3203.381923] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3203.536948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3207.904734] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3213.804041] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.814182] LustreError: Skipped 1 previous similar message [ 3215.249750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3215.799031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3219.975046] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3227.319240] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3232.332473] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3237.643539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3243.348091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3243.409979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3243.415983] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3243.566993] LustreError: 140-5: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3243.574156] LustreError: 81040:0:(mgs_handler.c:503:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3243.595424] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3243.601376] LustreError: 83260:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -98 [ 3243.606472] LustreError: 83260:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 3243.615912] LustreError: 83260:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 3243.718825] Lustre: server umount lustre-OST0000 complete [ 3243.722557] LustreError: 83260:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3248.337014] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3248.506421] LustreError: 140-5: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3248.512716] LustreError: 81040:0:(mgs_handler.c:503:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3248.524582] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3248.546087] LustreError: 83625:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -98 [ 3248.554894] LustreError: 83625:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 3248.561793] LustreError: 83625:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 3248.701481] Lustre: server umount lustre-OST0000 complete [ 3248.704600] LustreError: 83625:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3249.499647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3254.892595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3255.046730] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3255.051620] Lustre: Found index 0 for lustre-OST0000, updating log [ 3255.057775] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3255.116785] Lustre: lustre-OST0000: new disk, initializing [ 3255.120254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3255.317368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.714223] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3261.812253] Lustre: Failing over lustre-OST0000 [ 3261.955870] Lustre: server umount lustre-OST0000 complete [ 3265.512974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.519083] Lustre: Skipped 2 previous similar messages [ 3265.524112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3265.530683] Lustre: Skipped 4 previous similar messages [ 3268.642841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3268.658291] Lustre: Skipped 1 previous similar message [ 3273.697703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3273.709911] Lustre: Skipped 1 previous similar message [ 3278.816495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3278.820818] Lustre: Skipped 1 previous similar message [ 3279.327127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3279.466494] Lustre: server umount lustre-MDT0000 complete [ 3282.787859] LustreError: 83280:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124498 with bad export cookie 7120615748966571223 [ 3282.793389] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3282.797104] LustreError: 83280:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3291.223558] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:15:05 (1761124505) [ 3306.644228] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3317.058195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.431558] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3317.533988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.953228] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3330.673475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.560250] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3345.911213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3348.612514] Lustre: lustre-OST0000: precreate FID 0x0:4295077314 is over 4295077314 higher than LAST_ID 0x0:0, only precreating the last 10000 objects. OST replaced or reformatted? [ 3352.348122] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3360.657268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3360.737627] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3360.743273] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3360.793148] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3360.850175] Lustre: lustre-OST0001: new disk, initializing [ 3360.852976] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3361.085412] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3361.094086] Lustre: Skipped 2 previous similar messages [ 3365.721755] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3366.511616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3366.531893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3373.816475] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3381.103584] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 3388.884165] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.26@tcp (stopping) [ 3389.921308] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3389.941568] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.957229] Lustre: Skipped 1 previous similar message [ 3394.403715] Lustre: server umount lustre-MDT0001 complete [ 3394.406687] Lustre: Skipped 1 previous similar message [ 3395.560582] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3395.581146] LustreError: Skipped 4 previous similar messages [ 3397.545316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3404.356404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.474302] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3404.477071] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3404.521796] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3405.807714] LustreError: 90180:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0001: already exists, won't add [ 3405.813400] LustreError: 90180:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3405.819928] LustreError: 90180:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3405.827921] LustreError: 86412:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3405.930756] LustreError: 90190:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.201.126@tcp: cfg command failed: rc = -17 [ 3405.940519] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3405.940519] [ 3409.279713] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3410.415236] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 3410.426963] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3418.619907] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 05:17:12 (1761124632) [ 3434.453502] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3468.123833] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3471.639236] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3537.889311] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3537.900626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.915028] Lustre: Skipped 2 previous similar messages [ 3537.921700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3537.928419] Lustre: Skipped 5 previous similar messages [ 3543.601615] Lustre: server umount lustre-MDT0000 complete [ 3546.739821] LustreError: 89493:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124762 with bad export cookie 7120615748966572133 [ 3546.749276] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3546.753707] LustreError: 89493:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3555.807788] Lustre: 94075:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761124765/real 1761124765] req@00000000da8752be x1846672641647040/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761124771 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3565.535167] Lustre: 94276:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761124774/real 1761124774] req@0000000004a814d2 x1846672641647424/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761124780 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3570.538299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3577.895319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3578.653880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3585.159665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3592.855274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3593.838836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3609.705341] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3617.346625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.357721] Lustre: lustre-MDT0000: reset Object Index mappings [ 3617.678543] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3617.690982] LustreError: Skipped 5 previous similar messages [ 3617.726590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3617.732565] Lustre: Skipped 1 previous similar message [ 3620.770152] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3627.821193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.839944] Lustre: lustre-MDT0001: reset Object Index mappings [ 3632.525859] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3639.765147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3644.257528] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3644.460133] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:194 to 0x280000400:225 [ 3650.549150] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4295077475 to 0x0:4295077506 [ 3651.866439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.531502] Lustre: lustre-OST0001: deleting orphan objects from 0x0:162 to 0x0:193 [ 3653.544685] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:194 to 0x2c0000400:225 [ 3656.203275] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3663.626082] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3669.790866] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3719.914915] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:22:14 (1761124934) [ 3736.035588] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3736.051710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.073247] Lustre: Skipped 3 previous similar messages [ 3736.082082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3736.086020] Lustre: Skipped 8 previous similar messages [ 3749.343177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3749.501980] Lustre: server umount lustre-MDT0000 complete [ 3749.505943] Lustre: Skipped 3 previous similar messages [ 3749.859677] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3749.873150] LustreError: Skipped 9 previous similar messages [ 3752.465327] LustreError: 97189:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761124967 with bad export cookie 7120615748966627496 [ 3752.466876] LustreError: 166-1: MGC192.168.201.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3752.473665] LustreError: 97189:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3754.978301] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.989353] Lustre: Skipped 4 previous similar messages [ 3782.422101] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 3789.364866] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3796.810994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3801.898204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3806.823765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3812.366563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3818.304604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3818.391044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3818.584205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3818.622771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3818.692388] Lustre: lustre-MDT0000: new disk, initializing [ 3818.804664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3818.810499] Lustre: Skipped 3 previous similar messages [ 3818.833172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3822.381264] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3832.799185] Lustre: 104176:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761125042/real 1761125042] req@00000000236fff22 x1846672641698560/t0(0) o251->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761125048 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3834.685946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3838.882543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3838.939273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.206576] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3839.217238] Lustre: Modifying parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3842.471775] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_default_debug -1 all 8 [ 3852.255161] Lustre: 105277:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761125061/real 1761125061] req@000000006b6f9f13 x1846672641700864/t0(0) o251->MGC192.168.201.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761125067 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3869.707821] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing set_hostid [ 3875.779235] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing load_modules_local [ 3884.030238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3889.990347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3894.919898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3900.643703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3902.263861] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3762 sec ========== 05:25:16 (1761125116) [ 3935.571098] Lustre: DEBUG MARKER: oleg126-server.virtnet: executing unload_modules_local [ 3937.621909] Key type lgssc unregistered [ 3937.828166] LNet: 110128:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3937.843673] LNet: Removed LNI 192.168.201.126@tcp [ 3938.390700] Key type .llcrypt unregistered [ 3938.393194] Key type ._llcrypt unregistered