[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 596312581 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002016] kvm-guest: setup PV IPIs [ 0.005000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.005000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006012] pid_max: default: 32768 minimum: 301 [ 0.007131] LSM: Security Framework initializing [ 0.008055] Yama: becoming mindful. [ 0.009040] SELinux: Initializing. [ 0.010065] *** VALIDATE selinux *** [ 0.019159] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024513] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027116] *** VALIDATE tmpfs *** [ 0.029125] *** VALIDATE proc *** [ 0.030225] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.033113] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036082] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.039321] debug: unmapping init [mem 0xffffffff8da59000-0xffffffff8da60fff] [ 0.041150] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042655] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043022] ... version: 2 [ 0.044012] ... bit width: 48 [ 0.045010] ... generic registers: 4 [ 0.045965] ... value mask: 0000ffffffffffff [ 0.046011] ... max period: 00007fffffffffff [ 0.047011] ... fixed-purpose events: 3 [ 0.048011] ... event mask: 000000070000000f [ 0.049295] rcu: Hierarchical SRCU implementation. [ 0.051408] smp: Bringing up secondary CPUs ... [ 0.052579] x86: Booting SMP configuration: [ 0.053029] .... node #0, CPUs: #1 #2 #3 [ 0.063016] smp: Brought up 1 node, 4 CPUs [ 0.065019] smpboot: Max logical packages: 1 [ 0.066011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.094020] node 0 deferred pages initialised in 25ms [ 0.101000] devtmpfs: initialized [ 0.101000] x86/mm: Memory block size: 128MB [ 0.102000] gcov: version magic: 0x41383552 [ 0.106891] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.109120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.111305] pinctrl core: initialized pinctrl subsystem [ 0.113160] [ 0.113657] ************************************************************* [ 0.116012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.118008] ** ** [ 0.119010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.122010] ** ** [ 0.123010] ** This means that this kernel is built to expose internal ** [ 0.125008] ** IOMMU data structures, which may compromise security on ** [ 0.127008] ** your system. ** [ 0.129012] ** ** [ 0.131017] ** If you see this message and you are not debugging the ** [ 0.133010] ** kernel, report this immediately to your vendor! ** [ 0.135012] ** ** [ 0.136010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138010] ************************************************************* [ 0.140669] NET: Registered protocol family 16 [ 0.141520] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.144049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.146060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.149137] cpuidle: using governor menu [ 0.153955] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.157470] PCI: Using configuration type 1 for base access [ 0.158000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.174065] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.176018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.181469] cryptd: max_cpu_qlen set to 1000 [ 0.184362] ACPI: Added _OSI(Module Device) [ 0.188008] ACPI: Added _OSI(Processor Device) [ 0.189000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.189000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194039] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.199788] ACPI: Interpreter enabled [ 0.201060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.202016] ACPI: Using IOAPIC for interrupt routing [ 0.204096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209698] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223233] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238301] acpiphp: Slot [2] registered [ 0.240173] acpiphp: Slot [3] registered [ 0.242101] acpiphp: Slot [4] registered [ 0.244083] acpiphp: Slot [5] registered [ 0.245110] acpiphp: Slot [6] registered [ 0.247155] acpiphp: Slot [7] registered [ 0.249130] acpiphp: Slot [8] registered [ 0.251115] acpiphp: Slot [9] registered [ 0.253149] acpiphp: Slot [10] registered [ 0.256129] acpiphp: Slot [11] registered [ 0.257092] acpiphp: Slot [12] registered [ 0.258169] acpiphp: Slot [13] registered [ 0.260116] acpiphp: Slot [14] registered [ 0.262118] acpiphp: Slot [15] registered [ 0.264129] acpiphp: Slot [16] registered [ 0.266143] acpiphp: Slot [17] registered [ 0.268117] acpiphp: Slot [18] registered [ 0.270125] acpiphp: Slot [19] registered [ 0.272089] acpiphp: Slot [20] registered [ 0.275125] acpiphp: Slot [21] registered [ 0.277141] acpiphp: Slot [22] registered [ 0.279675] acpiphp: Slot [23] registered [ 0.281171] acpiphp: Slot [24] registered [ 0.283241] acpiphp: Slot [25] registered [ 0.285108] acpiphp: Slot [26] registered [ 0.287100] acpiphp: Slot [27] registered [ 0.288145] acpiphp: Slot [28] registered [ 0.289127] acpiphp: Slot [29] registered [ 0.291114] acpiphp: Slot [30] registered [ 0.293135] acpiphp: Slot [31] registered [ 0.294055] PCI host bridge to bus 0000:00 [ 0.296016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.299019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.301170] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.306017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.309035] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.312025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.314217] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.320077] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.325380] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.337018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.342053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.345018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.351015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.354027] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.355000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.358042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.360695] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.365015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.378857] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.383019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.388000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398030] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.403016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.421016] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.435049] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.449014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.467025] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.477000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.484015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.492015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.511016] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.524151] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.530017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.536015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.549021] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.556801] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.563014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.568014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.588016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.599212] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.605015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.610015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.632017] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.648301] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.651400] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.655713] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.659411] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.663249] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.669052] iommu: Default domain type: Passthrough [ 0.670000] SCSI subsystem initialized [ 0.671152] ACPI: bus type USB registered [ 0.673085] usbcore: registered new interface driver usbfs [ 0.674076] usbcore: registered new interface driver hub [ 0.677096] usbcore: registered new device driver usb [ 0.679223] pps_core: LinuxPPS API ver. 1 registered [ 0.682012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.685058] PTP clock support registered [ 0.689859] EDAC MC: Ver: 3.0.0 [ 0.692671] PCI: Using ACPI for IRQ routing [ 0.693759] NetLabel: Initializing [ 0.695018] NetLabel: domain hash size = 128 [ 0.697012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.698927] NetLabel: unlabeled traffic allowed by default [ 0.701643] vgaarb: loaded [ 0.704168] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.706010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.715000] clocksource: Switched to clocksource kvm-clock [ 0.913066] VFS: Disk quotas dquot_6.6.0 [ 0.915891] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.920647] *** VALIDATE ramfs *** [ 0.922131] *** VALIDATE hugetlbfs *** [ 0.924566] pnp: PnP ACPI init [ 0.927328] pnp: PnP ACPI: found 6 devices [ 0.952914] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.957140] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.960963] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.963893] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.968043] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.971276] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.974610] NET: Registered protocol family 2 [ 0.977696] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.985452] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.990640] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.996931] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.001180] TCP: Hash tables configured (established 65536 bind 65536) [ 1.004522] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.011636] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.016320] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.019522] NET: Registered protocol family 1 [ 1.023215] RPC: Registered named UNIX socket transport module. [ 1.025679] RPC: Registered udp transport module. [ 1.027950] RPC: Registered tcp transport module. [ 1.029681] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.032026] NET: Registered protocol family 44 [ 1.033892] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.036761] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.039589] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.042790] PCI: CLS 0 bytes, default 64 [ 1.046871] Unpacking initramfs... [ 4.450160] debug: unmapping init [mem 0xffff94987cc54000-0xffff94987ffbffff] [ 4.466612] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.468263] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.481398] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.249092] Initialise system trusted keyrings [ 6.250491] Key type blacklist registered [ 6.266357] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.310947] zbud: loaded [ 6.326989] *** VALIDATE nfs *** [ 6.335328] *** VALIDATE nfs4 *** [ 6.341812] pstore: using deflate compression [ 6.346270] Platform Keyring initialized [ 6.666653] NET: Registered protocol family 38 [ 6.668451] Key type asymmetric registered [ 6.677255] Asymmetric key parser 'x509' registered [ 6.680086] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.683098] io scheduler mq-deadline registered [ 6.684640] io scheduler kyber registered [ 6.686187] io scheduler bfq registered [ 6.693220] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.695951] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.706781] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.714659] ACPI: Power Button [PWRF] [ 7.136118] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.489948] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 8.092989] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 8.448887] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 9.351571] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 9.447139] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.521181] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.545913] Non-volatile memory driver v1.3 [ 9.547307] Linux agpgart interface v0.103 [ 9.628940] virtio_blk virtio1: [vda] 67992 512-byte logical blocks (34.8 MB/33.2 MiB) [ 9.636106] vda: detected capacity change from 0 to 34811904 [ 9.675595] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.697917] vdb: detected capacity change from 0 to 1073741824 [ 9.740021] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.748639] vdc: detected capacity change from 0 to 2621440000 [ 9.792840] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.806717] vdd: detected capacity change from 0 to 2621440000 [ 9.833348] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.835350] vde: detected capacity change from 0 to 4294967296 [ 9.886358] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.888727] vdf: detected capacity change from 0 to 4294967296 [ 9.915447] libphy: Fixed MDIO Bus: probed [ 9.931660] usbcore: registered new interface driver usbserial_generic [ 9.933516] usbserial: USB Serial support registered for generic [ 9.935354] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.969405] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.971782] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.984753] mousedev: PS/2 mouse device common for all mice [ 9.987942] rtc_cmos 00:05: RTC can wake from S4 [ 9.995935] rtc_cmos 00:05: registered as rtc0 [ 10.010484] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 10.012788] intel_pstate: CPU model not supported [ 10.018630] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 10.021691] hid: raw HID events driver (C) Jiri Kosina [ 10.034136] usbcore: registered new interface driver usbhid [ 10.036147] usbhid: USB HID core driver [ 10.038722] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 10.043230] drop_monitor: Initializing network drop monitor service [ 10.045687] Initializing XFRM netlink socket [ 10.047503] NET: Registered protocol family 10 [ 10.051316] Segment Routing with IPv6 [ 10.052607] NET: Registered protocol family 17 [ 10.054351] mpls_gso: MPLS GSO support [ 10.078323] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 10.087630] RAS: Correctable Errors collector initialized. [ 10.089503] AVX version of gcm_enc/dec engaged. [ 10.091393] AES CTR mode by8 optimization enabled [ 10.452108] sched_clock: Marking stable (10452087951, 0)->(11885986449, -1433898498) [ 10.463237] registered taskstats version 1 [ 10.468337] Loading compiled-in X.509 certificates [ 10.470914] zswap: loaded using pool lzo/zbud [ 10.515344] Key type big_key registered [ 10.552122] Key type encrypted registered [ 10.555991] ima: No TPM chip found, activating TPM-bypass! [ 10.557414] ima: Allocated hash algorithm: sha1 [ 10.558637] ima: No architecture policies found [ 10.560302] evm: Initialising EVM extended attributes: [ 10.561853] evm: security.selinux [ 10.562772] evm: security.ima [ 10.563658] evm: security.capability [ 10.564777] evm: HMAC attrs: 0x1 [ 10.582210] rtc_cmos 00:05: setting system clock to 2025-10-24 16:02:59 UTC (1761321779) [ 10.597381] debug: unmapping init [mem 0xffffffff8ea03000-0xffffffff8ebfffff] [ 10.611287] debug: unmapping init [mem 0xffffffff8d782000-0xffffffff8da58fff] [ 10.623053] Write protecting the kernel read-only data: 28672k [ 10.628886] debug: unmapping init [mem 0xffffffff8be03000-0xffffffff8bffffff] [ 10.634943] debug: unmapping init [mem 0xffffffff8c714000-0xffffffff8c7fffff] [ 10.708629] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.735892] systemd[1]: Detected virtualization kvm. [ 10.737481] systemd[1]: Detected architecture x86-64. [ 10.739274] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.813897] systemd[1]: No hostname configured. [ 10.815396] systemd[1]: Set hostname to . [ 10.817452] random: systemd: uninitialized urandom read (16 bytes read) [ 10.826690] systemd[1]: Initializing machine ID from random generator. [ 11.106881] random: systemd: uninitialized urandom read (16 bytes read) [ 11.109786] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 11.116246] random: systemd: uninitialized urandom read (16 bytes read) [ 11.132269] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 11.149280] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Swap. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.586666] device-mapper: uevent: version 1.0.3 [ 12.592443] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 14.347252] virtio_net virtio0 ens2: renamed from eth0 [ 14.404176] random: fast init done [ 14.869128] scsi host0: ata_piix [ 15.431778] scsi host1: ata_piix [ 15.433559] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 15.439611] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 20.107518] random: crng init done [ 20.108843] random: 7 urandom warning(s) missed due to ratelimiting [ 22.699575] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 25.132658] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.695866] printk: systemd: 25 output lines suppressed due to ratelimiting [ 28.596736] SELinux: Disabled at runtime. [ 28.708551] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.719186] systemd[1]: Detected virtualization kvm. [ 28.721760] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.231273] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.242505] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.265880] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.275106] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.281409] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.301367] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.322994] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ 30.437700] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 31.567878] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.640331] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.681155] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 34.365216] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.286198] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 39.530506] Key type dns_resolver registered [ ***] A start job is running for Configur…-only root support (9s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit)[ 40.623662] NFS: Registering the id_resolver key type [ 40.636546] Key type id_resolver registered [ 40.644148] Key type id_legacy registered [ *] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started OpenSSH server daemon. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg421-server login: [ 72.489832] libcfs: loading out-of-tree module taints kernel. [ 72.509114] alg: No test for adler32 (adler32-zlib) [ 73.261450] Key type ._llcrypt registered [ 73.263051] Key type .llcrypt registered [ 73.308600] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 83.503341] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 84.278403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.645496] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 85.066522] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 85.069082] LNet: Accept secure, port 988 [ 86.768323] Key type lgssc registered [ 87.494203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.864474] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 99.347169] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.662397] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.060351] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.925597] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 124.764874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.806424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 125.985513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 126.023201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 126.084501] Lustre: lustre-MDT0000: new disk, initializing [ 126.150706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.164852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 128.645912] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 136.501246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.538631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 136.543439] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 136.675950] Lustre: lustre-OST0000: new disk, initializing [ 136.678893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 136.682574] Lustre: Skipped 1 previous similar message [ 136.712295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.931553] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 146.630971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 146.671462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 146.675607] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 146.732711] Lustre: lustre-OST0001: new disk, initializing [ 146.735656] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 146.765289] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 148.879746] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 155.516544] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 161.564379] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 169.131978] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing check_logdir /tmp/testlogs/ [ 171.998494] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing yml_node [ 175.035399] Lustre: DEBUG MARKER: Client: 2.15.7.12 [ 176.390815] Lustre: DEBUG MARKER: MDS: 2.15.7.12 [ 177.644241] Lustre: DEBUG MARKER: OSS: 2.15.7.12 [ 178.499910] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 12:05:46 EDT 2025 [ 182.967388] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 183.864136] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 200.675392] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 200.677517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 200.690342] Lustre: Skipped 1 previous similar message [ 203.236936] Lustre: server umount lustre-MDT0000 complete [ 205.731710] LustreError: 8363:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321974 with bad export cookie 11835334027226487061 [ 205.732680] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 205.742507] LustreError: 8363:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 205.785297] Lustre: server umount lustre-OST0000 complete [ 207.842742] Lustre: server umount lustre-OST0001 complete [ 211.285431] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 215.413860] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 221.093931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.537250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 227.697696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 231.152201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 231.189659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.301633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 231.320091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 231.364743] Lustre: lustre-MDT0000: new disk, initializing [ 231.417582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.433314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 233.442625] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 238.372149] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 241.434399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 241.476282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.481496] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 241.610236] Lustre: lustre-OST0000: new disk, initializing [ 241.615995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 241.619127] Lustre: Skipped 1 previous similar message [ 241.660357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 243.530242] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 248.862266] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 250.844187] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 250.946705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 252.897670] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 252.900729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 252.910592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 252.913166] Lustre: Skipped 1 previous similar message [ 258.017310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 263.139213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 266.208289] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 266.301579] Lustre: server umount lustre-OST0000 complete [ 274.402180] Lustre: 14606:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322037/real 1761322037] req@00000000732646cc x1846879823357120/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322043 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 274.488284] Lustre: server umount lustre-MDT0000 complete [ 276.603097] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 12:07:25 (1761322045) [ 279.622783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 279.859730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 281.684444] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 284.618291] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 287.454779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 289.346825] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 292.344629] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 299.513778] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 304.834799] Lustre: server umount lustre-MDT0000 complete [ 306.313467] LustreError: 15112:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322075 with bad export cookie 11835334027226488468 [ 306.318163] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.800197] Lustre: 17069:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322075/real 1761322075] req@0000000066028bce x1846879823363008/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322081 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 319.581940] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 323.441471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.657689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 323.661112] Lustre: Skipped 1 previous similar message [ 325.324246] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 328.772075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 330.670694] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 334.226834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 334.263139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 334.266987] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 334.339100] Lustre: lustre-OST0001: new disk, initializing [ 334.341936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 336.265546] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 341.263648] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 342.746563] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 354.273713] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.280585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.283568] Lustre: Skipped 1 previous similar message [ 358.434098] Lustre: server umount lustre-MDT0000 complete [ 358.436279] Lustre: Skipped 1 previous similar message [ 360.340274] LustreError: 20665:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322129 with bad export cookie 11835334027226488986 [ 360.342588] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.346115] LustreError: 20665:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 368.492892] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 372.902286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.130471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.133059] Lustre: Skipped 2 previous similar messages [ 375.201188] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 379.236610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 381.420252] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 385.145436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 386.881402] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 395.599155] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 397.091447] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 397.094174] Lustre: Skipped 1 previous similar message [ 403.937200] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.938101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 403.941652] Lustre: Skipped 2 previous similar messages [ 403.946727] Lustre: Skipped 1 previous similar message [ 410.067121] Lustre: server umount lustre-MDT0000 complete [ 410.069052] Lustre: Skipped 2 previous similar messages [ 411.638171] LustreError: 22581:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322180 with bad export cookie 11835334027226489798 [ 411.639839] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 411.644252] LustreError: 22581:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 416.538526] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 12:09:45 (1761322185) [ 425.573711] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 430.195436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.268228] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 435.852548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 437.724347] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 441.534312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.619549] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 441.622424] Lustre: Skipped 4 previous similar messages [ 443.252979] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 452.021801] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 453.591561] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 461.285124] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.293586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 461.297039] Lustre: Skipped 3 previous similar messages [ 468.261075] LustreError: 27225:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322237 with bad export cookie 11835334027226490470 [ 468.262914] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 468.266693] LustreError: 27225:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 473.427568] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 12:10:42 (1761322242) [ 478.526802] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 482.503339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.439992] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 487.777784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 487.905677] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.564759] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 493.008517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 493.026673] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.851207] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 498.322590] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 499.693472] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 508.386172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.386880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 508.391552] Lustre: Skipped 2 previous similar messages [ 508.396122] Lustre: Skipped 3 previous similar messages [ 512.523309] Lustre: server umount lustre-MDT0000 complete [ 512.525314] Lustre: Skipped 5 previous similar messages [ 516.046684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.128603] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.134700] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xa43f8d976aeda914 to 0xa43f8d976aedabd7 [ 516.141405] Lustre: MGC192.168.204.121@tcp: Connection restored to 192.168.204.121@tcp (at 0@lo) [ 516.304938] BUG: spinlock bad magic on CPU#2, lctl/34302 [ 516.307264] lock: 0xffff9498d01cb900, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 516.309851] CPU: 2 PID: 34302 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 516.314066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 516.316797] Call Trace: [ 516.317689] ? dump_stack+0xbb/0x10e [ 516.318782] ? spin_dump.cold.3+0x32/0x65 [ 516.320004] ? do_raw_spin_lock+0xce/0x190 [ 516.320848] ? _raw_spin_lock+0x21/0x30 [ 516.322011] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 516.324155] ? __seccomp_filter+0x4b/0x680 [ 516.327007] ? full_proxy_write+0x5e/0xa0 [ 516.328483] ? __vfs_write+0x1c/0x60 [ 516.330590] ? vfs_write+0xd8/0x2b0 [ 516.331674] ? ksys_write+0x66/0x120 [ 516.332923] ? __x64_sys_write+0x1e/0x30 [ 516.333808] ? do_syscall_64+0xc1/0x440 [ 516.334635] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 518.090147] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 521.098242] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 521.774253] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 531.071496] LustreError: 31364:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322299 with bad export cookie 11835334027226491863 [ 531.076408] LustreError: 31364:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 536.109217] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 12:11:44 (1761322304) [ 541.462940] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 545.737451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.739427] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 551.341560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 551.397452] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.403085] LustreError: Skipped 1 previous similar message [ 553.463286] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 557.026490] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.078688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 559.066301] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 562.644539] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 564.012916] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 564.015521] Lustre: Skipped 1 previous similar message [ 577.133409] LustreError: 36850:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322346 with bad export cookie 11835334027226492325 [ 577.139176] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 577.143803] LustreError: Skipped 1 previous similar message [ 583.648252] Lustre: 39449:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322346/real 1761322346] req@00000000c8e67eee x1846879823424512/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761322352 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 584.160174] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.893657] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 12:12:37 (1761322357) [ 589.887297] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 590.586242] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 12:12:39 (1761322359) [ 595.021918] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 598.202679] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 602.555938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 605.167260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 607.967696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 607.998468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.117064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 608.160294] Lustre: lustre-MDT0000: new disk, initializing [ 608.197285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.200554] Lustre: Skipped 7 previous similar messages [ 608.210191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 610.022302] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 614.617429] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 617.337274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 617.362694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 617.366548] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 619.318671] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 623.811932] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 624.587785] Lustre: DEBUG MARKER: create test files [ 634.337789] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 634.338047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.348497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 634.350932] Lustre: Skipped 3 previous similar messages [ 647.136207] Lustre: 44135:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322410/real 1761322410] req@00000000e4823948 x1846879823432000/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322416 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 647.147204] Lustre: 44135:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 647.231894] Lustre: server umount lustre-MDT0000 complete [ 647.234026] Lustre: Skipped 7 previous similar messages [ 652.572980] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 653.881594] Key type lgssc unregistered [ 654.045429] LNet: 44737:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.048645] LNet: Removed LNI 192.168.204.121@tcp [ 654.430183] Key type .llcrypt unregistered [ 654.431694] Key type ._llcrypt unregistered [ 666.359636] alg: No test for adler32 (adler32-zlib) [ 667.111439] Key type ._llcrypt registered [ 667.113091] Key type .llcrypt registered [ 667.157681] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 667.636476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.807737] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 667.903644] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 667.906245] LNet: Accept secure, port 988 [ 669.496231] Key type lgssc registered [ 669.918426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 673.006538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.203317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 675.812079] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 678.701146] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 681.386699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 681.521830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 683.279061] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 686.447274] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 689.700756] Lustre: lustre-OST0000: deleting orphan objects from 0x0:102 to 0x0:129 [ 689.731303] Lustre: 47577:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 689.736081] Lustre: 47577:0:(osd_handler.c:1947:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 689.739328] Lustre: 47577:0:(osd_handler.c:1954:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 689.742869] Lustre: 47577:0:(osd_handler.c:1964:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 689.746756] Lustre: 47577:0:(osd_handler.c:1971:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 689.750185] Lustre: 47577:0:(osd_handler.c:1978:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 694.958584] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 696.910499] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 699.873721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 699.874103] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 699.878840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 704.993541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 705.133414] Lustre: server umount lustre-OST0000 complete [ 712.160220] Lustre: 48587:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322475/real 1761322475] req@0000000091361da3 x1846880434656512/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322481 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 712.237216] Lustre: server umount lustre-MDT0000 complete [ 717.595971] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 718.764204] Key type lgssc unregistered [ 718.902459] LNet: 49192:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.906142] LNet: Removed LNI 192.168.204.121@tcp [ 719.240636] Key type .llcrypt unregistered [ 719.242139] Key type ._llcrypt unregistered [ 731.929739] alg: No test for adler32 (adler32-zlib) [ 732.684393] Key type ._llcrypt registered [ 732.685926] Key type .llcrypt registered [ 732.737493] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 733.235223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.387497] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 733.477476] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 733.480158] LNet: Accept secure, port 988 [ 735.072174] Key type lgssc registered [ 735.513620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 738.786281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.991154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 741.653751] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 744.496115] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 747.008104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 747.122166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 748.570587] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 751.128538] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 755.300656] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 755.321987] Lustre: 52129:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 755.325228] Lustre: 52129:0:(osd_handler.c:1947:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 755.327849] Lustre: 52129:0:(osd_handler.c:1954:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 755.330324] Lustre: 52129:0:(osd_handler.c:1964:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 755.333511] Lustre: 52129:0:(osd_handler.c:1971:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 755.336385] Lustre: 52129:0:(osd_handler.c:1978:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 759.416484] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 765.921312] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 765.921329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.931368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 767.971329] Lustre: server umount lustre-OST0000 complete [ 775.648261] Lustre: 53090:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322538/real 1761322538] req@0000000026d2f6dc x1846880503862336/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322544 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 775.727100] Lustre: server umount lustre-MDT0000 complete [ 780.664033] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 781.708522] Key type lgssc unregistered [ 781.827382] LNet: 53693:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.830685] LNet: Removed LNI 192.168.204.121@tcp [ 782.138476] Key type .llcrypt unregistered [ 782.139866] Key type ._llcrypt unregistered [ 787.732020] alg: No test for adler32 (adler32-zlib) [ 788.483354] Key type ._llcrypt registered [ 788.484858] Key type .llcrypt registered [ 788.527957] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 794.493711] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 794.922501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 795.057084] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 795.133856] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 795.136204] LNet: Accept secure, port 988 [ 796.712152] Key type lgssc registered [ 797.074651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 800.222664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 802.351195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 804.552671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 807.093060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 807.122493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.221573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 808.236143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 808.277195] Lustre: lustre-MDT0000: new disk, initializing [ 808.309912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.318784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 809.922364] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 814.042642] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 816.587892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 816.623574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 816.627970] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 816.725177] Lustre: lustre-OST0000: new disk, initializing [ 816.727794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 816.730406] Lustre: Skipped 1 previous similar message [ 816.753971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 818.252739] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 822.166195] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 823.720581] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 823.803769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 827.362131] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 827.362481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.370422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 830.679084] Lustre: server umount lustre-OST0000 complete [ 837.088189] Lustre: 58880:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322600/real 1761322600] req@00000000560a8f19 x1846880568874816/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322606 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 837.159511] Lustre: server umount lustre-MDT0000 complete [ 840.225123] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 12:16:48 (1761322608) [ 843.462691] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 848.134645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 855.771798] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 858.561337] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 861.853254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 863.850681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 865.947247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 868.303742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 868.331484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.412425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 868.424589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 868.460882] Lustre: lustre-MDT0000: new disk, initializing [ 868.485946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 868.493222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 869.978626] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 873.805544] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 876.036326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 876.060578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 876.063296] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 876.151271] Lustre: lustre-OST0000: new disk, initializing [ 876.152941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 876.154735] Lustre: Skipped 1 previous similar message [ 876.173080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.580467] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 881.606334] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 883.326845] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 883.414689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 886.172725] Lustre: server umount lustre-OST0000 complete [ 893.920196] Lustre: 65197:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322656/real 1761322656] req@00000000d55e7f49 x1846880568880256/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761322662 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 900.064139] Lustre: 65197:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322662/real 1761322662] req@00000000d55e7f49 x1846880568880704/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322668 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 900.129553] Lustre: server umount lustre-MDT0000 complete [ 903.137835] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 12:17:51 (1761322671) [ 905.663985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.840947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.284161] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 909.863609] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 912.020920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 913.598166] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 916.121418] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 917.451094] Lustre: 67642:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting 6adf0d06-f803-4869-bf46-93be0cd4fc76 at adminstrative request [ 917.473293] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 917.477705] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 923.617409] Lustre: MGS: Client 6adf0d06-f803-4869-bf46-93be0cd4fc76 (at 0@lo) reconnecting [ 923.621242] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xbefb4e0d1027d642 to 0xbefb4e0d1027d799 [ 923.626899] Lustre: MGC192.168.204.121@tcp: Connection restored to 192.168.204.121@tcp (at 0@lo) [ 949.982425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 950.008505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 950.012061] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 950.057343] Lustre: lustre-OST0001: new disk, initializing [ 950.059548] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 950.082841] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 950.085622] Lustre: Skipped 1 previous similar message [ 951.573827] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 955.422131] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 956.469669] Lustre: server umount lustre-OST0001 complete [ 957.932433] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 962.528764] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 962.531605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.535439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 963.041347] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.160936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.161376] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.162661] Lustre: Skipped 1 previous similar message [ 972.256080] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 972.323764] Lustre: server umount lustre-OST0000 complete [ 979.936207] Lustre: 69240:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322742/real 1761322742] req@00000000ee3944ee x1846880568893440/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322748 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 982.759571] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 12:19:11 (1761322751) [ 983.289911] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 983.901785] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 12:19:12 (1761322752) [ 984.450693] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 985.088827] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 12:19:13 (1761322753) [ 985.628805] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 986.266387] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 12:19:14 (1761322754) [ 986.865146] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 988.709509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 989.066462] loop: module loaded [ 989.071198] loop0: detected capacity change from 0 to 4294967296 [ 989.083585] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 989.574020] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 990.721605] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 992.047809] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 994.774347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 996.066256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 997.412177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 999.814083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 999.837235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 999.839549] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 999.849184] LustreError: 71667:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 999.851390] LustreError: 71667:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 999.853921] LustreError: 71667:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 999.857434] LustreError: 71667:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1004.699183] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 1007.348684] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1010.790528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1012.699317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1014.734479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1017.054162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1017.077947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.160529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1017.171817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1017.210474] Lustre: lustre-MDT0000: new disk, initializing [ 1017.241376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.249281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1018.687810] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1022.563715] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1024.840284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1024.865991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1024.870106] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1026.483073] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1030.312530] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1032.018816] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1032.105943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1034.854713] Lustre: server umount lustre-OST0000 complete [ 1034.857036] Lustre: Skipped 1 previous similar message [ 1042.400146] Lustre: 76437:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322805/real 1761322805] req@00000000c9711e00 x1846880568899328/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761322811 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1056.730019] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 12:20:24 (1761322824) [ 1062.727212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.144894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 1063.147836] Lustre: Skipped 1 previous similar message [ 1066.744440] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1073.250830] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1079.149435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1082.907014] hrtimer: interrupt took 3451757 ns [ 1083.144105] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1090.918518] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1097.684493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1097.745396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1097.751447] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1097.867846] Lustre: lustre-OST0001: new disk, initializing [ 1097.875879] Lustre: Skipped 1 previous similar message [ 1097.886383] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1097.898619] Lustre: Skipped 2 previous similar messages [ 1101.495732] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1108.195731] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1120.479476] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1128.877883] Lustre: Failing over lustre-MDT0000 [ 1130.260602] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 1130.978107] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.985785] Lustre: Skipped 1 previous similar message [ 1131.233100] Lustre: server umount lustre-MDT0000 complete [ 1131.237225] Lustre: Skipped 1 previous similar message [ 1143.264382] Lustre: 55039:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322905/real 1761322905] req@00000000e11be1b3 x1846880568913472/t0(0) o400->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322912 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1143.275883] Lustre: 55039:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1143.279542] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1148.900796] LDISKFS-fs (dm-0): recovery complete [ 1148.903333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.409869] Lustre: Evicted from MGS (at 192.168.204.121@tcp) after server handle changed from 0xbefb4e0d1027daef to 0xbefb4e0d10286735 [ 1149.416682] Lustre: MGC192.168.204.121@tcp: Connection restored to (at 0@lo) [ 1149.656863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 1149.660043] Lustre: Skipped 2 previous similar messages [ 1149.686299] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1151.964734] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1161.929156] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1161.933315] Lustre: 81268:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1162.021683] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1162.344132] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1162.666769] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1162.674253] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 7 previous similar messages [ 1162.688463] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1162.693283] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1163.010384] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1163.016736] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 1163.940802] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1163.963911] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1163.976426] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 3 previous similar messages [ 1163.985218] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1163.996507] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 1164.312543] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1164.319073] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 1166.200961] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1166.205750] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 6 previous similar messages [ 1166.209379] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1166.214366] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1166.529242] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1166.532740] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 1170.306710] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1170.313577] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 1170.320895] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1170.328206] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 1170.648221] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1170.651127] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 1178.506449] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1178.512668] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 25 previous similar messages [ 1178.515573] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1178.519264] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 1178.832173] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1178.835688] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 1194.819853] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1194.824632] Lustre: 81302:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 51 previous similar messages [ 1194.828796] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 1194.833245] LustreError: 81302:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 51 previous similar messages [ 1195.152257] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 1195.157325] LustreError: 81302:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 51 previous similar messages [ 1221.232165] Lustre: 81302:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1221.236626] Lustre: 81302:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1221.240469] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1221.242910] LustreError: 81302:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 1221.248234] Lustre: 81302:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@0000000039348f0b x1846880563654720/t0(8589934785) o101->514b0daf-1ac3-4d7d-9129-f1d2a21b1d61@192.168.204.21@tcp:105/0 lens 592/0 e 0 to 0 dl 1761322995 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 1221.262175] Lustre: lustre-MDT0000: Recovery over after 1:00, of 2 clients 1 recovered and 1 was evicted. [ 1221.283673] Lustre: lustre-OST0000: deleting orphan objects from 0x0:81 to 0x0:577 [ 1221.284042] Lustre: lustre-OST0001: deleting orphan objects from 0x0:49 to 0x0:545 [ 1225.699873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.722413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1225.727589] Lustre: Skipped 3 previous similar messages [ 1227.475711] Lustre: server umount lustre-OST0000 complete [ 1230.818464] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.819290] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.937050] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.942176] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1235.947689] Lustre: Skipped 1 previous similar message [ 1244.640178] Lustre: 82429:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323007/real 1761323007] req@000000006508e927 x1846880568924480/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323013 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1250.495681] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 12:23:38 (1761323018) [ 1255.032995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1255.037260] Lustre: *** cfs_fail_loc=197, val=0*** [ 1272.609190] LustreError: 83121:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 1: rc = -5 [ 1283.872402] LustreError: 83121:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 1290.036770] Lustre: *** cfs_fail_loc=197, val=0*** [ 1290.039104] Lustre: Skipped 65 previous similar messages [ 1313.632724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.203355] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1320.885623] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1327.332612] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 12:24:55 (1761323095) [ 1344.207820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.582131] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1351.106176] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1359.840174] Lustre: 85465:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323122/real 1761323122] req@00000000bd796a89 x1846880568928128/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323128 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1359.939920] Lustre: server umount lustre-MDT0000 complete [ 1359.942389] Lustre: Skipped 3 previous similar messages [ 1365.581100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1372.368954] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 1376.435211] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1381.271499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1384.508730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1387.480204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1390.762471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1390.794370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.918749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1390.941475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1390.986943] Lustre: lustre-MDT0000: new disk, initializing [ 1391.039893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1392.850382] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1397.781822] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1400.860368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1400.900705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1400.904907] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1403.044419] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1407.596634] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1409.463664] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1409.558537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1414.625658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.625906] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1414.631675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1424.352242] Lustre: 90855:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323187/real 1761323187] req@0000000039bfa0ea x1846880568934400/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323193 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1428.346513] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 12:26:36 (1761323196) [ 1433.218800] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1434.458404] Key type lgssc unregistered [ 1434.590836] LNet: 91658:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.595501] LNet: Removed LNI 192.168.204.121@tcp [ 1434.971604] Key type .llcrypt unregistered [ 1434.973481] Key type ._llcrypt unregistered [ 1442.294200] alg: No test for adler32 (adler32-zlib) [ 1443.046391] Key type ._llcrypt registered [ 1443.047940] Key type .llcrypt registered [ 1443.098110] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 1450.802035] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1451.370527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1451.551294] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1451.678372] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1451.682675] LNet: Accept secure, port 988 [ 1453.312958] Key type lgssc registered [ 1453.816426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1458.391512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1461.418331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1464.415164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1467.112996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1470.437165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1470.475751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.601366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1471.630137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1471.674875] Lustre: lustre-MDT0000: new disk, initializing [ 1471.739232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.753214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1473.736923] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1479.649449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1479.683388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1479.687696] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1479.809272] Lustre: lustre-OST0000: new disk, initializing [ 1479.813187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1479.817414] Lustre: Skipped 1 previous similar message [ 1479.846500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.212200] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1488.501923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1488.551937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1488.555908] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1488.631974] Lustre: lustre-OST0001: new disk, initializing [ 1488.635322] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1488.688259] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1490.949308] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1500.129740] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.130448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1500.135328] Lustre: Skipped 1 previous similar message [ 1500.141121] Lustre: Skipped 1 previous similar message [ 1502.552347] Lustre: server umount lustre-MDT0000 complete [ 1506.109827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1508.524035] LustreError: 94726:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323277 with bad export cookie 6251614621646631330 [ 1508.526533] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1508.538196] LustreError: 94726:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1508.594923] Lustre: server umount lustre-OST0000 complete [ 1510.779911] Lustre: server umount lustre-OST0001 complete [ 1514.597032] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 12:28:03 (1761323283) [ 1518.343937] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1522.014741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1522.598900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1524.956261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1525.392120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1527.854487] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 12:28:16 (1761323296) [ 1532.126920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1533.183376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1534.072363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1534.923836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1535.336603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1536.192881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1543.253154] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 1547.486302] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1551.958700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1554.855117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1557.945041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1561.404727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1561.442521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.551617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1561.567089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1561.607930] Lustre: lustre-MDT0000: new disk, initializing [ 1561.649412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.659346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1563.782508] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1568.825575] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1572.303780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1572.356381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1572.360976] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1572.484720] Lustre: lustre-OST0000: new disk, initializing [ 1572.490646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1572.496235] Lustre: Skipped 1 previous similar message [ 1572.536316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.651922] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1579.882567] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1582.020689] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1582.136847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1586.658293] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1586.660919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.673645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1589.290597] Lustre: server umount lustre-OST0000 complete [ 1597.408513] Lustre: 105015:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323360/real 1761323360] req@00000000291b3d1a x1846881256752576/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323366 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1597.499135] Lustre: server umount lustre-MDT0000 complete [ 1601.181763] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 12:29:29 (1761323369) [ 1604.165448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.440475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.622229] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1610.264957] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1613.376521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1615.497815] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1619.056463] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1626.594050] Lustre: *** cfs_fail_loc=159, val=0*** [ 1626.596017] Lustre: Skipped 1 previous similar message [ 1633.045751] Lustre: lustre-MDT0000: Client e71eab25-2f25-451b-a35d-3cdd215a0ec5 (at 192.168.204.21@tcp) reconnecting [ 1633.503629] Lustre: *** cfs_fail_loc=159, val=0*** [ 1633.506987] Lustre: Skipped 5 previous similar messages [ 1640.718970] Lustre: lustre-MDT0000: Client e71eab25-2f25-451b-a35d-3cdd215a0ec5 (at 192.168.204.21@tcp) reconnecting [ 1645.538048] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1645.538694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.547672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1648.110725] Lustre: server umount lustre-OST0000 complete [ 1655.776237] Lustre: 107768:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323418/real 1761323418] req@00000000ee3a56cd x1846881256759040/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323424 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1661.303468] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1662.517548] Key type lgssc unregistered [ 1662.645507] LNet: 108370:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.649488] LNet: Removed LNI 192.168.204.121@tcp [ 1662.990256] Key type .llcrypt unregistered [ 1662.991845] Key type ._llcrypt unregistered [ 1673.578953] alg: No test for adler32 (adler32-zlib) [ 1674.330359] Key type ._llcrypt registered [ 1674.331709] Key type .llcrypt registered [ 1674.377306] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1674.754438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1674.984447] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1675.087032] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1675.089762] LNet: Accept secure, port 988 [ 1676.688213] Key type lgssc registered [ 1677.178775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1680.759224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1682.000607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1683.823664] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1686.747289] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1689.579837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1689.731400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.719781] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1694.901997] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1696.870713] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 1706.085381] Lustre: *** cfs_fail_loc=159, val=0*** [ 1713.429426] Lustre: lustre-MDT0000: Client f1568c96-5f12-4cdd-a0f5-a8e0cad795e0 (at 192.168.204.21@tcp) reconnecting [ 1714.100523] Lustre: *** cfs_fail_loc=159, val=0*** [ 1721.620796] Lustre: lustre-MDT0000: Client f1568c96-5f12-4cdd-a0f5-a8e0cad795e0 (at 192.168.204.21@tcp) reconnecting [ 1722.042063] Lustre: *** cfs_fail_loc=159, val=0*** [ 1722.044044] Lustre: Skipped 3 previous similar messages [ 1729.300697] Lustre: lustre-MDT0000: Client f1568c96-5f12-4cdd-a0f5-a8e0cad795e0 (at 192.168.204.21@tcp) reconnecting [ 1731.307484] Lustre: *** cfs_fail_loc=159, val=0*** [ 1731.310977] Lustre: Skipped 5 previous similar messages [ 1738.507485] Lustre: lustre-MDT0000: Client 0f825bd2-3912-4baf-b0e2-bb614c9771b7 (at 192.168.204.21@tcp) reconnecting [ 1738.989430] Lustre: *** cfs_fail_loc=159, val=0*** [ 1738.991464] Lustre: Skipped 12 previous similar messages [ 1746.182504] Lustre: lustre-MDT0000: Client 0f825bd2-3912-4baf-b0e2-bb614c9771b7 (at 192.168.204.21@tcp) reconnecting [ 1748.449168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.449282] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1748.455968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1753.569756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1754.338340] Lustre: server umount lustre-OST0000 complete [ 1761.248151] Lustre: 112445:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323524/real 1761323524] req@00000000939e088a x1846881490575424/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323530 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1761.329665] Lustre: server umount lustre-MDT0000 complete [ 1766.512648] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1767.595130] Key type lgssc unregistered [ 1767.720337] LNet: 113047:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.724408] LNet: Removed LNI 192.168.204.121@tcp [ 1768.042095] Key type .llcrypt unregistered [ 1768.043980] Key type ._llcrypt unregistered [ 1777.976666] alg: No test for adler32 (adler32-zlib) [ 1778.728381] Key type ._llcrypt registered [ 1778.729646] Key type .llcrypt registered [ 1778.777499] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1779.264796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1779.419161] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1779.526336] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1779.528764] LNet: Accept secure, port 988 [ 1781.120140] Key type lgssc registered [ 1781.561693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1785.184434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.392302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.165703] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1791.335139] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1794.042202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1794.180731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.015379] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1799.491468] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1801.326194] Lustre: lustre-OST0000: deleting orphan objects from 0x0:91 to 0x0:129 [ 1809.593400] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1810.925283] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 1810.928756] Lustre: Skipped 1 previous similar message [ 1814.764757] Lustre: server umount lustre-OST0000 complete [ 1822.688240] Lustre: 116784:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323585/real 1761323585] req@000000000031ca82 x1846881600674496/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761323591 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1828.832189] Lustre: 116784:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323591/real 1761323591] req@000000001435d2fd x1846881600674944/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323597 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1828.913545] Lustre: server umount lustre-MDT0000 complete [ 1834.628853] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1835.889920] Key type lgssc unregistered [ 1836.028710] LNet: 117387:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.036425] LNet: Removed LNI 192.168.204.121@tcp [ 1836.435661] Key type .llcrypt unregistered [ 1836.438102] Key type ._llcrypt unregistered [ 1846.690317] alg: No test for adler32 (adler32-zlib) [ 1847.442394] Key type ._llcrypt registered [ 1847.444033] Key type .llcrypt registered [ 1847.490122] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1847.858128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1848.106628] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1848.199680] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1848.202843] LNet: Accept secure, port 988 [ 1849.800815] Key type lgssc registered [ 1850.276729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1853.912435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.132671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.951472] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1860.354294] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1863.176819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1863.333151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.100756] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1868.189749] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1870.501316] Lustre: lustre-OST0000: deleting orphan objects from 0x0:91 to 0x0:161 [ 1876.710756] Lustre: *** cfs_fail_loc=159, val=0*** [ 1882.899567] Lustre: lustre-MDT0000: Client d3fef25b-18ef-44ef-8b2b-ef739b7728be (at 192.168.204.21@tcp) reconnecting [ 1886.177627] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1886.177768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.185548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1890.162843] Lustre: server umount lustre-OST0000 complete [ 1897.952448] Lustre: 120839:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323660/real 1761323660] req@00000000b3855fa0 x1846881673025280/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323666 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1898.029344] Lustre: server umount lustre-MDT0000 complete [ 1904.018606] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1905.170114] Key type lgssc unregistered [ 1905.293625] LNet: 121443:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.297304] LNet: Removed LNI 192.168.204.121@tcp [ 1905.639518] Key type .llcrypt unregistered [ 1905.641631] Key type ._llcrypt unregistered [ 1915.963208] alg: No test for adler32 (adler32-zlib) [ 1916.715325] Key type ._llcrypt registered [ 1916.719282] Key type .llcrypt registered [ 1916.770859] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1917.205271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1917.357043] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1917.455944] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1917.458434] LNet: Accept secure, port 988 [ 1919.048230] Key type lgssc registered [ 1919.482690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1922.680476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.899702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.670939] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1928.608436] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1931.272393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1931.394824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.030692] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 1935.900483] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1939.558151] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 1955.081413] Lustre: lustre-OST0000: evicting b0823536-74b8-4744-9fbf-4ebbe395c4ec (at 192.168.204.21@tcp) by administrative request [ 1980.898123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.898202] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1980.904600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1985.767542] Lustre: server umount lustre-OST0000 complete [ 1993.696351] Lustre: 125319:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323756/real 1761323756] req@0000000019b3875b x1846881745379456/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323762 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1993.777468] Lustre: server umount lustre-MDT0000 complete [ 1999.540499] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2000.760147] Key type lgssc unregistered [ 2000.898738] LNet: 125922:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.902285] LNet: Removed LNI 192.168.204.121@tcp [ 2001.268665] Key type .llcrypt unregistered [ 2001.270323] Key type ._llcrypt unregistered [ 2027.837540] alg: No test for adler32 (adler32-zlib) [ 2028.589388] Key type ._llcrypt registered [ 2028.591023] Key type .llcrypt registered [ 2028.637109] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2028.986588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2029.218766] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2029.298630] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2029.300948] LNet: Accept secure, port 988 [ 2030.896139] Key type lgssc registered [ 2031.306576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2034.178333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.332908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.794715] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2039.381593] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2041.783904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2041.915642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.479364] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2046.336128] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2051.044852] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 2056.161823] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2056.162421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.171113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2057.576799] Lustre: server umount lustre-OST0000 complete [ 2065.376602] Lustre: 130340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323828/real 1761323828] req@00000000767e45d5 x1846881862816896/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323834 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2065.449519] Lustre: server umount lustre-MDT0000 complete [ 2070.481938] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2071.562963] Key type lgssc unregistered [ 2071.677347] LNet: 130942:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.680631] LNet: Removed LNI 192.168.204.121@tcp [ 2071.969265] Key type .llcrypt unregistered [ 2071.970373] Key type ._llcrypt unregistered [ 2081.244768] alg: No test for adler32 (adler32-zlib) [ 2081.996469] Key type ._llcrypt registered [ 2081.997957] Key type .llcrypt registered [ 2082.038859] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2082.449641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.619603] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2082.709181] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2082.712149] LNet: Accept secure, port 988 [ 2084.304138] Key type lgssc registered [ 2084.728148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2087.539160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2090.076470] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 12:37:38 (1761323858) [ 2094.775239] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 2097.671416] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2101.088528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2103.119199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2105.350951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2107.740802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2107.760846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.863354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2108.876289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2108.914880] Lustre: lustre-MDT0000: new disk, initializing [ 2108.946195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.954702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2110.623716] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2114.774247] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2117.246858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2117.269521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2117.272990] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2117.365318] Lustre: lustre-OST0000: new disk, initializing [ 2117.367920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2117.370621] Lustre: Skipped 1 previous similar message [ 2117.395894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.871474] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2122.853912] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2131.937272] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2131.937443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.945945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2137.057191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2137.638276] Lustre: server umount lustre-OST0000 complete [ 2144.224157] Lustre: 137092:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323907/real 1761323907] req@000000005a97ff95 x1846881918392576/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323913 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2144.290638] Lustre: server umount lustre-MDT0000 complete [ 2149.280787] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2150.327743] Key type lgssc unregistered [ 2150.442345] LNet: 137696:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.445522] LNet: Removed LNI 192.168.204.121@tcp [ 2150.745393] Key type .llcrypt unregistered [ 2150.747357] Key type ._llcrypt unregistered [ 2159.925954] alg: No test for adler32 (adler32-zlib) [ 2160.678395] Key type ._llcrypt registered [ 2160.679777] Key type .llcrypt registered [ 2160.723266] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2161.087971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2161.322831] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2161.407423] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2161.409912] LNet: Accept secure, port 988 [ 2162.992109] Key type lgssc registered [ 2163.374537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2166.256667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.440779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.054970] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2172.037377] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2174.812420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2174.952730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.638933] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2179.583279] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2183.079340] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2187.694381] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2187.701543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.707215] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2187.710879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2187.715994] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2187.717224] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2188.083661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2188.089724] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 2188.439150] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2188.442462] Lustre: Skipped 1 previous similar message [ 2188.448308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.453154] Lustre: Skipped 1 previous similar message [ 2188.454967] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2188.457423] Lustre: Skipped 1 previous similar message [ 2188.458905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2188.463206] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2188.463548] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153 to 0x0:193 [ 2188.465498] Lustre: Skipped 1 previous similar message [ 2188.794242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2188.801722] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153 to 0x0:225 [ 2189.145622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2189.150471] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153 to 0x0:257 [ 2189.482112] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2189.485222] Lustre: Skipped 2 previous similar messages [ 2189.493255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.498747] Lustre: Skipped 2 previous similar messages [ 2189.501321] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2189.504161] Lustre: Skipped 2 previous similar messages [ 2189.506785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2189.511978] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2189.512447] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153 to 0x0:289 [ 2189.515660] Lustre: Skipped 2 previous similar messages [ 2189.878711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2189.884215] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153 to 0x0:321 [ 2190.229117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2190.235443] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153 to 0x0:353 [ 2190.580508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2190.587073] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153 to 0x0:385 [ 2190.923419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2190.928050] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153 to 0x0:417 [ 2193.377444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.380921] Lustre: Skipped 4 previous similar messages [ 2193.382554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2198.497067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2199.530758] Lustre: server umount lustre-OST0000 complete [ 2206.176162] Lustre: 141842:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323969/real 1761323969] req@0000000020bd9e80 x1846882001231808/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323975 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2206.251160] Lustre: server umount lustre-MDT0000 complete [ 2211.468451] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2212.634309] Key type lgssc unregistered [ 2212.753691] LNet: 142444:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.757325] LNet: Removed LNI 192.168.204.121@tcp [ 2213.099100] Key type .llcrypt unregistered [ 2213.101378] Key type ._llcrypt unregistered [ 2223.141660] alg: No test for adler32 (adler32-zlib) [ 2223.893389] Key type ._llcrypt registered [ 2223.895336] Key type .llcrypt registered [ 2223.939105] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2224.276793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.547646] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2224.645938] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2224.648642] LNet: Accept secure, port 988 [ 2226.240216] Key type lgssc registered [ 2226.682657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2229.760208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.962445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2232.652677] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2235.733675] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2238.376966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2238.510315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.068603] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2242.848222] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2246.694369] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153 to 0x0:449 [ 2251.289493] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2252.000380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.006649] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2252.010449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2252.016325] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.121@tcp (at 0@lo) [ 2252.016448] LustreError: 143923:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2254.611109] Lustre: lustre-OST0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 2256.865754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.871742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2259.713207] Lustre: lustre-OST0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 2261.985297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2267.104153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2267.104987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2267.113230] Lustre: Skipped 1 previous similar message [ 2267.179084] Lustre: server umount lustre-OST0000 complete [ 2272.096122] LustreError: 143923:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 2272.099420] LustreError: 143923:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2295.581721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2295.704311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.270691] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2300.028133] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2317.179087] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 16 sec [ 2318.179220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2318.183109] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.121@tcp (at 0@lo) [ 2318.185050] Lustre: lustre-OST0000: deleting orphan objects from 0x0:153 to 0x0:481 [ 2319.147245] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a26054e4800.ost_server_uuid 40 [ 2319.628566] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a26054e4800.ost_server_uuid in FULL state after 0 sec [ 2323.425500] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.425729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.434113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2326.689488] Lustre: server umount lustre-OST0000 complete [ 2333.152096] Lustre: 147444:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324096/real 1761324096] req@00000000694eefce x1846882067297216/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324102 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2333.216668] Lustre: server umount lustre-MDT0000 complete [ 2338.002910] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2339.114433] Key type lgssc unregistered [ 2339.257317] LNet: 148047:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.260428] LNet: Removed LNI 192.168.204.121@tcp [ 2339.573778] Key type .llcrypt unregistered [ 2339.575269] Key type ._llcrypt unregistered [ 2350.979206] alg: No test for adler32 (adler32-zlib) [ 2351.731419] Key type ._llcrypt registered [ 2351.732821] Key type .llcrypt registered [ 2351.782258] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2352.127347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2352.348979] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2352.434150] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2352.436248] LNet: Accept secure, port 988 [ 2354.024117] Key type lgssc registered [ 2354.388309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2357.604135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.760722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.231222] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2363.033391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2363.138703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.197587] Lustre: lustre-OST0000: deleting orphan objects from 0x0:483 to 0x0:513 [ 2364.506665] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2367.283598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2367.306484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2367.310379] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2367.352894] Lustre: lustre-OST0001: new disk, initializing [ 2367.354445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2367.377056] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2368.686612] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2374.135452] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2380.953192] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2395.105089] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.105685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2395.108647] Lustre: Skipped 1 previous similar message [ 2399.221756] Lustre: server umount lustre-MDT0000 complete [ 2400.524076] LustreError: 149749:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324169 with bad export cookie 6474190443764957040 [ 2400.526250] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2400.527313] LustreError: 149749:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2400.543790] Lustre: server umount lustre-OST0000 complete [ 2401.839813] Lustre: server umount lustre-OST0001 complete [ 2402.923659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2402.950754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2403.316689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2403.349283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2403.727022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2403.752665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2407.925691] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2411.352337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.540278] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.080961] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2416.343198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2416.492565] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.506918] LustreError: 137-5: mylustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.510554] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:545 [ 2418.107483] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2421.330570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2422.985952] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2426.265195] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2427.549398] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2437.089781] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2437.090416] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2437.094055] Lustre: Skipped 1 previous similar message [ 2437.098429] Lustre: Skipped 2 previous similar messages [ 2442.209640] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2442.212677] Lustre: Skipped 1 previous similar message [ 2443.066047] Lustre: server umount mylustre-MDT0000 complete [ 2444.364091] LustreError: 156315:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324213 with bad export cookie 6474190443764959175 [ 2444.365585] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2444.369173] LustreError: 156315:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2446.729269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2446.761295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2447.099807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2447.126474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2447.485787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2447.509143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2451.571201] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2454.812291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.967685] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.969809] Lustre: Skipped 1 previous similar message [ 2456.362365] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2459.291238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2460.451178] LustreError: 137-5: tfs-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.453678] Lustre: tfs-OST0000: deleting orphan objects from 0x0:516 to 0x0:577 [ 2460.823569] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2463.843153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2465.392250] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 2465.410457] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2468.524957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2469.680569] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2479.585174] Lustre: tfs-MDT0000-lwp-OST0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.585685] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 2479.589300] Lustre: Skipped 1 previous similar message [ 2482.606156] Lustre: server umount tfs-MDT0000 complete [ 2482.607749] Lustre: Skipped 2 previous similar messages [ 2483.775087] LustreError: 160579:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324252 with bad export cookie 6474190443764960372 [ 2483.776790] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.778776] LustreError: 160579:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2486.059107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2486.088251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2486.430440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2486.466365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2486.810286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2486.830212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2490.781989] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2493.951680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.096738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2494.098462] Lustre: Skipped 2 previous similar messages [ 2495.422728] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2498.109227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2499.234315] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.237272] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:609 [ 2499.518330] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2502.210926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2503.567409] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2504.173509] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 2506.544760] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2507.683023] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2514.757445] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 12:44:43 (1761324283) [ 2528.225714] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.226334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2528.229584] Lustre: Skipped 1 previous similar message [ 2528.231496] Lustre: Skipped 1 previous similar message [ 2532.599620] Lustre: server umount lustre-MDT0000 complete [ 2532.601570] Lustre: Skipped 2 previous similar messages [ 2533.896173] LustreError: 162558:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324302 with bad export cookie 6474190443764961611 [ 2533.897917] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2533.899866] LustreError: 162558:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2548.192129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2555.486054] Lustre: server umount lustre-OST0001 complete [ 2555.487391] Lustre: Skipped 1 previous similar message [ 2557.493707] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 2560.079220] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2563.321044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2565.165780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2567.142247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2569.238027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2569.254636] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 2569.254636] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 2569.254636] [ 2569.259738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 2569.329740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2569.337927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2569.373729] Lustre: lustre-MDT0000: new disk, initializing [ 2569.393309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.394925] Lustre: Skipped 2 previous similar messages [ 2569.399463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2570.682903] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2574.734900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2574.757641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2574.761178] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2574.850874] Lustre: lustre-OST0000: new disk, initializing [ 2574.852584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2574.854343] Lustre: Skipped 1 previous similar message [ 2576.383886] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2580.376805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2580.402210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2580.406170] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2580.451954] Lustre: lustre-OST0001: new disk, initializing [ 2580.453628] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2581.739709] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2585.739712] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 12:45:54 (1761324354) [ 2589.076543] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 12:45:57 (1761324357) [ 2601.954599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.958508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2601.960037] Lustre: Skipped 1 previous similar message [ 2606.578735] Lustre: server umount lustre-OST0000 complete [ 2607.073406] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.192952] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.527358] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2621.181247] LustreError: 167586:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324390 with bad export cookie 6474190443764962738 [ 2621.183148] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.185021] LustreError: 167586:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2629.351391] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 2635.020776] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2635.723667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2638.046426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2640.056651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2642.210131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2644.450614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2644.473230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.565255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2644.576775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2644.612346] Lustre: lustre-MDT0000: new disk, initializing [ 2644.638249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.640108] Lustre: Skipped 2 previous similar messages [ 2644.644845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2645.956261] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2649.595346] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2651.638381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2651.656812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2651.659722] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2652.945154] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2656.801209] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2663.393467] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2663.393552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.402862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2663.404444] Lustre: Skipped 1 previous similar message [ 2672.096146] Lustre: 175326:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324435/real 1761324435] req@0000000080cef356 x1846882201582208/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324441 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2672.159848] Lustre: server umount lustre-MDT0000 complete [ 2672.161366] Lustre: Skipped 3 previous similar messages [ 2677.073286] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2678.175562] Key type lgssc unregistered [ 2678.310438] LNet: 175929:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2678.313284] LNet: Removed LNI 192.168.204.121@tcp [ 2678.653223] Key type .llcrypt unregistered [ 2678.654416] Key type ._llcrypt unregistered [ 2688.371788] alg: No test for adler32 (adler32-zlib) [ 2689.123315] Key type ._llcrypt registered [ 2689.124186] Key type .llcrypt registered [ 2689.156528] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2689.496641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2689.694169] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2689.768551] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2689.770745] LNet: Accept secure, port 988 [ 2691.344128] Key type lgssc registered [ 2691.673761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2694.237832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.409455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.835921] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2699.335428] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2701.505594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2701.604588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.027150] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2705.656691] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2706.004108] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 2706.708405] Lustre: server umount lustre-OST0000 complete [ 2714.592169] Lustre: 179366:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324477/real 1761324477] req@0000000004685ed8 x1846882554876096/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324483 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2714.653856] Lustre: server umount lustre-MDT0000 complete [ 2718.251107] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2719.233760] Key type lgssc unregistered [ 2719.340319] LNet: 179967:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2719.342716] LNet: Removed LNI 192.168.204.121@tcp [ 2719.610315] Key type .llcrypt unregistered [ 2719.611318] Key type ._llcrypt unregistered [ 2725.546102] alg: No test for adler32 (adler32-zlib) [ 2726.297310] Key type ._llcrypt registered [ 2726.298206] Key type .llcrypt registered [ 2726.329340] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2726.635514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2726.843893] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2726.912577] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2726.913938] LNet: Accept secure, port 988 [ 2728.488149] Key type lgssc registered [ 2728.852592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2731.801136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2748.256280] LustreError: 181206:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 1: rc = -5 [ 2780.000236] LustreError: 181206:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 2810.720287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2812.170831] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2814.661280] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2815.435626] Lustre: server umount lustre-OST0000 complete [ 2820.060155] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2821.057025] Key type lgssc unregistered [ 2821.182394] LNet: 182692:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2821.185938] LNet: Removed LNI 192.168.204.121@tcp [ 2821.487601] Key type .llcrypt unregistered [ 2821.488930] Key type ._llcrypt unregistered [ 2833.167263] alg: No test for adler32 (adler32-zlib) [ 2833.918338] Key type ._llcrypt registered [ 2833.919839] Key type .llcrypt registered [ 2833.953845] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2834.372130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2834.612105] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2834.697599] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2834.698956] LNet: Accept secure, port 988 [ 2836.280155] Key type lgssc registered [ 2836.638875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2840.741867] loop0: detected capacity change from 0 to 204800000 [ 2840.794653] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2840.798708] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2840.923511] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2840.928242] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2840.952649] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2840.973610] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2840.991231] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2841.020172] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2841.049860] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2841.080609] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2841.425310] loop0: detected capacity change from 0 to 204800000 [ 2841.477527] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2841.712580] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2842.048528] loop0: detected capacity change from 0 to 204800000 [ 2842.194786] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2842.534982] loop0: detected capacity change from 0 to 204800000 [ 2842.685294] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2842.977617] loop0: detected capacity change from 0 to 204800000 [ 2842.989437] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2844.260840] loop0: detected capacity change from 0 to 204800000 [ 2844.279504] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2845.562119] loop0: detected capacity change from 0 to 204800000 [ 2845.570371] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2846.876534] loop0: detected capacity change from 0 to 204800000 [ 2846.885275] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2848.118817] loop0: detected capacity change from 0 to 204800000 [ 2848.154995] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2848.183060] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.202655] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2848.207266] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 2848.209846] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 2848.212568] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 2848.215683] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 2848.220286] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 2848.223548] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 2848.226919] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 2848.231745] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 2848.234655] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 2848.237766] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 2848.242180] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 2848.246471] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 2848.250601] Lustre: 185996:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 2856.928173] Lustre: 186275:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324619/real 1761324619] req@000000005943d566 x1846882706915456/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324625 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2856.945143] Lustre: server umount MGS complete [ 2857.289128] loop0: detected capacity change from 0 to 204800000 [ 2857.325636] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.459495] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.474879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.489194] LustreError: 186329:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2857.801092] loop1: detected capacity change from 0 to 204800000 [ 2857.866588] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2857.890763] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.907886] Lustre: 186443:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2857.910847] Lustre: 186443:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 2857.914074] Lustre: 186443:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 2857.919243] Lustre: 186443:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 2857.923829] Lustre: 186443:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 2857.928773] Lustre: 186443:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 2857.933528] Lustre: 186443:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 2857.941526] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2858.006797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2858.018872] LustreError: 186355:0:(tgt_lastrcvd.c:1503:tgt_last_rcvd_update()) lustre-MDT0000: trying to overwrite bigger transno:on-disk: 8589934594, new: 8589934593 replay: 0. See LU-617. [ 2858.032144] LustreError: 186443:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0001: Aborting recovery for device [ 2858.341996] loop2: detected capacity change from 0 to 204800000 [ 2858.369783] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2858.402850] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2858.406578] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 2858.424932] Lustre: 186528:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2858.428244] Lustre: 186528:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 2858.432992] Lustre: 186528:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 2858.435995] Lustre: 186528:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 2858.439526] Lustre: 186528:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 2858.443089] Lustre: 186528:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 2858.449426] Lustre: 186528:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 2858.506105] Lustre: Found index 0 for lustre-OST0000, updating log [ 2858.537655] LustreError: 186528:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 2858.860129] loop3: detected capacity change from 0 to 204800000 [ 2858.920901] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2858.943467] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2858.947642] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 2858.965347] Lustre: 186616:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 2858.970702] Lustre: 186616:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 2858.975832] Lustre: 186616:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 2858.981222] Lustre: 186616:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 2858.986120] Lustre: 186616:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 2858.992173] Lustre: 186616:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 2858.997087] Lustre: 186616:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 2859.031659] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2859.033780] LustreError: 186386:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 2859.034474] Lustre: Skipped 1 previous similar message [ 2859.044383] LustreError: 186616:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 2863.075101] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 2863.075460] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 2863.079277] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 2863.079847] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 2864.270672] LustreError: 186557:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 237us: rc = -2 [ 2878.432920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.433107] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2878.433936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2878.436163] Lustre: Skipped 1 previous similar message [ 2881.412531] Lustre: server umount lustre-MDT0000 complete [ 2881.736495] LustreError: 186337:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324650 with bad export cookie 969664191719430691 [ 2881.738293] LustreError: 166-1: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2881.740157] LustreError: 186337:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2883.552662] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.552775] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.557240] LustreError: Skipped 1 previous similar message [ 2883.557595] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2883.557600] Lustre: Skipped 3 previous similar messages [ 2883.560613] Lustre: Skipped 3 previous similar messages [ 2886.560538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2886.560879] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2886.562683] Lustre: Skipped 2 previous similar messages [ 2886.568870] LustreError: Skipped 1 previous similar message [ 2892.768499] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2892.768581] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.771193] Lustre: Skipped 3 previous similar messages [ 2892.778768] LustreError: Skipped 3 previous similar messages [ 2896.352115] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2896.445604] Lustre: server umount lustre-MDT0001 complete [ 2911.200115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2911.231895] Lustre: server umount lustre-OST0000 complete [ 2917.743945] Lustre: server umount lustre-OST0001 complete [ 2920.290198] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 12:51:29 (1761324689) [ 2927.594352] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 2930.407919] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2933.992330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2935.928026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2937.824720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2939.903557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2939.926336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.002180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2940.013557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2940.055067] Lustre: lustre-MDT0000: new disk, initializing [ 2940.077517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.084352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2941.436923] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2944.983879] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2947.103424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2947.124975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.127826] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2947.204588] Lustre: lustre-OST0000: new disk, initializing [ 2947.206458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2947.208535] Lustre: Skipped 1 previous similar message [ 2947.227751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.616580] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 2952.291809] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2954.605774] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 2960.921989] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 2970.512337] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 2978.061207] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3021.281443] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3021.281546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.289828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3031.521783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3031.523871] Lustre: Skipped 1 previous similar message [ 3034.080169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3034.151356] Lustre: server umount lustre-OST0000 complete [ 3041.760152] Lustre: 193537:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324804/real 1761324804] req@000000001c280505 x1846882706958528/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324810 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3044.762181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.137228] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3054.048150] Lustre: 194548:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324816/real 1761324816] req@00000000fe491fb5 x1846882706958656/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324822 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3054.061130] Lustre: server umount MGS complete [ 3054.062917] Lustre: Skipped 1 previous similar message [ 3057.708373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.843236] LustreError: 195182:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3057.860202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.171693] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3061.568570] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3063.724724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3065.147972] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3067.555048] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3094.497177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3094.497620] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3094.500725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3096.760196] Lustre: server umount lustre-OST0000 complete [ 3103.200116] Lustre: 197477:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324866/real 1761324866] req@000000008397c552 x1846882706968832/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324872 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 3107.678527] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3108.654053] Key type lgssc unregistered [ 3108.766440] LNet: 198079:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3108.769351] LNet: Removed LNI 192.168.204.121@tcp [ 3109.047439] Key type .llcrypt unregistered [ 3109.049192] Key type ._llcrypt unregistered [ 3119.398672] alg: No test for adler32 (adler32-zlib) [ 3120.150331] Key type ._llcrypt registered [ 3120.151825] Key type .llcrypt registered [ 3120.183862] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 3125.040288] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3125.449859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3125.587803] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3125.670423] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3125.672132] LNet: Accept secure, port 988 [ 3127.256076] Key type lgssc registered [ 3127.584633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3130.384117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3132.307057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3134.329183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3136.607976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3136.636838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.730588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3137.741052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3137.777963] Lustre: lustre-MDT0000: new disk, initializing [ 3137.802424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3137.810171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3139.248908] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3143.114471] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3145.353866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3145.379783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3145.383149] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3145.474685] Lustre: lustre-OST0000: new disk, initializing [ 3145.476620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3145.479184] Lustre: Skipped 1 previous similar message [ 3145.499763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3146.914445] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3150.518742] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3158.919830] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3165.114917] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 3173.430681] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 3227.106077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.106191] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3227.109399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3228.758351] Lustre: server umount lustre-OST0000 complete [ 3235.296127] Lustre: 204859:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324998/real 1761324998] req@000000005f652b2e x1846883012075136/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325004 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3235.360744] Lustre: server umount lustre-MDT0000 complete [ 3238.278355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.765982] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3247.584152] Lustre: 205869:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325010/real 1761325010] req@00000000ae9aa0bd x1846883012075264/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325016 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3247.594788] Lustre: server umount MGS complete [ 3251.460350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.610921] LustreError: 206504:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3251.631882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.061804] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3255.514285] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3257.649606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3257.747149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3259.220507] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3261.768626] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3282.401387] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3282.401684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.410094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3287.520828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3288.088284] Lustre: server umount lustre-OST0000 complete [ 3295.712109] Lustre: 208784:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325058/real 1761325058] req@00000000f6222be2 x1846883012085568/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325064 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3295.773681] Lustre: server umount lustre-MDT0000 complete [ 3300.772054] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3301.817845] Key type lgssc unregistered [ 3301.935382] LNet: 209386:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3301.939638] LNet: Removed LNI 192.168.204.121@tcp [ 3302.225195] Key type .llcrypt unregistered [ 3302.226417] Key type ._llcrypt unregistered [ 3312.083803] alg: No test for adler32 (adler32-zlib) [ 3312.835349] Key type ._llcrypt registered [ 3312.836304] Key type .llcrypt registered [ 3312.878176] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3313.487356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3313.644811] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3313.732138] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3313.735434] LNet: Accept secure, port 988 [ 3315.312150] Key type lgssc registered [ 3315.662225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3318.285423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.441313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.867705] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3323.280346] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3325.520822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3325.623295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.996285] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3329.463514] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3331.662913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3331.686667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3331.691484] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3331.741829] Lustre: lustre-OST0001: new disk, initializing [ 3331.744584] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3331.772466] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3333.143299] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3334.529046] LustreError: 213300:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3335.585680] LustreError: 213300:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3336.609913] LustreError: 213300:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3337.859861] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3337.943078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3338.659471] LustreError: 213300:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3338.662461] LustreError: 213300:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 3339.820556] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a2607889800.ost_server_uuid 40 [ 3340.282663] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a2607889800.ost_server_uuid in FULL state after 0 sec [ 3342.153369] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a2607889800.ost_server_uuid 40 [ 3342.691064] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a2607889800.ost_server_uuid in FULL state after 0 sec [ 3342.754879] LustreError: 213300:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 3342.758754] LustreError: 213300:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 3356.131305] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3356.135958] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3356.935223] Lustre: lustre-OST0001: Not available for connect from 192.168.204.21@tcp (stopping) [ 3360.304794] Lustre: server umount lustre-OST0001 complete [ 3361.250121] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.370183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.370439] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.371001] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3366.374864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3367.901379] Lustre: server umount lustre-OST0000 complete [ 3375.584141] Lustre: 214815:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325138/real 1761325138] req@00000000b88672b6 x1846883209197184/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325144 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3375.649374] Lustre: server umount lustre-MDT0000 complete [ 3380.293263] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3381.276736] Key type lgssc unregistered [ 3381.384386] LNet: 215417:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3381.388871] LNet: Removed LNI 192.168.204.121@tcp [ 3381.693557] Key type .llcrypt unregistered [ 3381.695179] Key type ._llcrypt unregistered [ 3393.716656] alg: No test for adler32 (adler32-zlib) [ 3394.468296] Key type ._llcrypt registered [ 3394.469346] Key type .llcrypt registered [ 3394.507077] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3394.824684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3395.054415] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3395.136984] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3395.138552] LNet: Accept secure, port 988 [ 3396.712150] Key type lgssc registered [ 3397.097250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3399.869310] loop0: detected capacity change from 0 to 1099511627776 [ 3400.023251] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 3400.027912] *** VALIDATE xfs *** [ 3400.032636] XFS (loop0): Mounting V5 Filesystem [ 3400.041653] XFS (loop0): Ending clean mount [ 3400.470550] loop1: detected capacity change from 0 to 18691697672192 [ 3416.758490] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3421.587480] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 3424.551352] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3427.903151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3429.811205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3431.846199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3434.254467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3434.281520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.371216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3435.380816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3435.418199] Lustre: lustre-MDT0000: new disk, initializing [ 3435.441228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3435.447591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3436.888339] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3440.677214] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3443.008075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3443.040978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3443.044115] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3443.142246] Lustre: lustre-OST0000: new disk, initializing [ 3443.144337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3443.146249] Lustre: Skipped 1 previous similar message [ 3443.170711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3444.706447] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3448.747029] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3450.248790] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3450.330624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3454.433439] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3454.435852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.440970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3457.178367] Lustre: server umount lustre-OST0000 complete [ 3464.672154] Lustre: 222110:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325227/real 1761325227] req@000000004bdb6287 x1846883295172608/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325233 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3464.737450] Lustre: server umount lustre-MDT0000 complete [ 3467.371268] XFS (loop0): Unmounting Filesystem [ 3469.031138] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 13:00:37 (1761325237) [ 3471.217564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.381964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.732877] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3475.197812] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3477.403927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3477.501667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.972645] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all 8 [ 3481.844394] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3491.809339] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3491.809593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.817453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3493.532513] Lustre: server umount lustre-OST0000 complete [ 3501.024161] Lustre: 224922:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325263/real 1761325263] req@00000000a13dae6e x1846883295177472/t0(0) o251->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325269 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3501.091349] Lustre: server umount lustre-MDT0000 complete [ 3505.649140] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3506.568971] Key type lgssc unregistered [ 3506.671367] LNet: 225525:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3506.674352] LNet: Removed LNI 192.168.204.121@tcp [ 3506.951792] Key type .llcrypt unregistered [ 3506.952795] Key type ._llcrypt unregistered [ 3516.765710] alg: No test for adler32 (adler32-zlib) [ 3517.517372] Key type ._llcrypt registered [ 3517.518350] Key type .llcrypt registered [ 3517.551771] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 3522.721781] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3523.047847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3523.270110] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3523.348282] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3523.350765] LNet: Accept secure, port 988 [ 3524.936083] Key type lgssc registered [ 3525.281769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3528.634190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3530.776149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3532.981827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3533.534615] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3355 sec ========== 13:01:42 (1761325302) [ 3543.933737] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3544.914133] Key type lgssc unregistered [ 3545.028373] LNet: 230117:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3545.032591] LNet: Removed LNI 192.168.204.121@tcp [ 3545.342422] Key type .llcrypt unregistered [ 3545.343541] Key type ._llcrypt unregistered