[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 525225166 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003145] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.006374] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009160] LSM: Security Framework initializing [ 0.010072] Yama: becoming mindful. [ 0.011045] SELinux: Initializing. [ 0.012073] *** VALIDATE selinux *** [ 0.021663] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026491] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027180] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029171] *** VALIDATE tmpfs *** [ 0.031488] *** VALIDATE proc *** [ 0.033286] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.036296] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040036] Spectre V2 : User space: Vulnerable [ 0.041015] Speculative Store Bypass: Vulnerable [ 0.044807] debug: unmapping init [mem 0xffffffff90859000-0xffffffff90860fff] [ 0.047274] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048674] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049030] ... version: 2 [ 0.050013] ... bit width: 48 [ 0.051016] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053018] ... max period: 00007fffffffffff [ 0.054021] ... fixed-purpose events: 3 [ 0.055019] ... event mask: 000000070000000f [ 0.056387] rcu: Hierarchical SRCU implementation. [ 0.059162] smp: Bringing up secondary CPUs ... [ 0.060728] x86: Booting SMP configuration: [ 0.061053] .... node #0, CPUs: #1 #2 #3 [ 0.067406] smp: Brought up 1 node, 4 CPUs [ 0.069022] smpboot: Max logical packages: 1 [ 0.070026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.160546] node 0 deferred pages initialised in 88ms [ 0.165109] devtmpfs: initialized [ 0.167321] x86/mm: Memory block size: 128MB [ 0.170164] gcov: version magic: 0x41383552 [ 0.173196] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.176082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.178363] pinctrl core: initialized pinctrl subsystem [ 0.181185] [ 0.181724] ************************************************************* [ 0.184014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186013] ** ** [ 0.188012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.191015] ** ** [ 0.193011] ** This means that this kernel is built to expose internal ** [ 0.196014] ** IOMMU data structures, which may compromise security on ** [ 0.199011] ** your system. ** [ 0.201009] ** ** [ 0.204012] ** If you see this message and you are not debugging the ** [ 0.206014] ** kernel, report this immediately to your vendor! ** [ 0.208015] ** ** [ 0.210010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.212009] ************************************************************* [ 0.214810] NET: Registered protocol family 16 [ 0.216502] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.218054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.221063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.225048] cpuidle: using governor menu [ 0.227922] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.228789] PCI: Using configuration type 1 for base access [ 0.232162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.249124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.253039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.257134] cryptd: max_cpu_qlen set to 1000 [ 0.263515] ACPI: Added _OSI(Module Device) [ 0.266058] ACPI: Added _OSI(Processor Device) [ 0.267013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.278042] ACPI: Added _OSI(Processor Aggregator Device) [ 0.282741] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.292747] ACPI: Interpreter enabled [ 0.296117] ACPI: PM: (supports S0 S3 S4 S5) [ 0.298020] ACPI: Using IOAPIC for interrupt routing [ 0.300187] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.305421] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.318468] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.321049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.323026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.326081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.333000] acpiphp: Slot [2] registered [ 0.336200] acpiphp: Slot [5] registered [ 0.338159] acpiphp: Slot [6] registered [ 0.340149] acpiphp: Slot [7] registered [ 0.341138] acpiphp: Slot [8] registered [ 0.344384] acpiphp: Slot [9] registered [ 0.348181] acpiphp: Slot [10] registered [ 0.351176] acpiphp: Slot [3] registered [ 0.353099] acpiphp: Slot [4] registered [ 0.355097] acpiphp: Slot [11] registered [ 0.359264] acpiphp: Slot [12] registered [ 0.362679] acpiphp: Slot [13] registered [ 0.366184] acpiphp: Slot [14] registered [ 0.367000] acpiphp: Slot [15] registered [ 0.367000] acpiphp: Slot [16] registered [ 0.367219] acpiphp: Slot [17] registered [ 0.368000] acpiphp: Slot [18] registered [ 0.368000] acpiphp: Slot [19] registered [ 0.368170] acpiphp: Slot [20] registered [ 0.369000] acpiphp: Slot [21] registered [ 0.369000] acpiphp: Slot [22] registered [ 0.369000] acpiphp: Slot [23] registered [ 0.369000] acpiphp: Slot [24] registered [ 0.369040] acpiphp: Slot [25] registered [ 0.370000] acpiphp: Slot [26] registered [ 0.370000] acpiphp: Slot [27] registered [ 0.371000] acpiphp: Slot [28] registered [ 0.371000] acpiphp: Slot [29] registered [ 0.371000] acpiphp: Slot [30] registered [ 0.371000] acpiphp: Slot [31] registered [ 0.371000] PCI host bridge to bus 0000:00 [ 0.372028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.373000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.373034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.374000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.374026] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.375000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.375224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.376000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.376000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.381000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.382068] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.383000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.383027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.384029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.385602] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.386000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.386000] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387728] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.389000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.392000] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.393000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.397473] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.404022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.410020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.430020] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.441736] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.475019] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.487230] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.498031] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.511036] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.532017] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.545277] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.555019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.561018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.584032] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.598314] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.616150] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.633019] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.645000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.650017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.655020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.671019] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.678000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.678000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.678000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.680325] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.682231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.689031] iommu: Default domain type: Passthrough [ 0.692333] SCSI subsystem initialized [ 0.693138] ACPI: bus type USB registered [ 0.694104] usbcore: registered new interface driver usbfs [ 0.695000] usbcore: registered new interface driver hub [ 0.695000] usbcore: registered new device driver usb [ 0.695000] pps_core: LinuxPPS API ver. 1 registered [ 0.697011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.700060] PTP clock support registered [ 0.704058] EDAC MC: Ver: 3.0.0 [ 0.707464] PCI: Using ACPI for IRQ routing [ 0.709228] NetLabel: Initializing [ 0.711020] NetLabel: domain hash size = 128 [ 0.713009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.715083] NetLabel: unlabeled traffic allowed by default [ 0.717320] vgaarb: loaded [ 0.719808] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.720000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.728000] clocksource: Switched to clocksource kvm-clock [ 0.848883] VFS: Disk quotas dquot_6.6.0 [ 0.850511] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.853076] *** VALIDATE ramfs *** [ 0.854221] *** VALIDATE hugetlbfs *** [ 0.855402] pnp: PnP ACPI init [ 0.857606] pnp: PnP ACPI: found 6 devices [ 0.874559] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.877538] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.879343] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.881574] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.883638] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.886010] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.888589] NET: Registered protocol family 2 [ 0.890937] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.895970] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.899451] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.904541] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.907779] TCP: Hash tables configured (established 65536 bind 65536) [ 0.910797] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.913695] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.916572] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.920358] NET: Registered protocol family 1 [ 0.923872] RPC: Registered named UNIX socket transport module. [ 0.926614] RPC: Registered udp transport module. [ 0.928035] RPC: Registered tcp transport module. [ 0.929503] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.931560] NET: Registered protocol family 44 [ 0.933368] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.935705] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.937553] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.939796] PCI: CLS 0 bytes, default 64 [ 0.941406] Unpacking initramfs... [ 2.729080] debug: unmapping init [mem 0xffff93153cc54000-0xffff93153ffbffff] [ 2.735660] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.738055] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.742578] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.330603] Initialise system trusted keyrings [ 3.332277] Key type blacklist registered [ 3.335895] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.344386] zbud: loaded [ 3.348397] *** VALIDATE nfs *** [ 3.349891] *** VALIDATE nfs4 *** [ 3.352213] pstore: using deflate compression [ 3.355633] Platform Keyring initialized [ 3.539547] NET: Registered protocol family 38 [ 3.543910] Key type asymmetric registered [ 3.546284] Asymmetric key parser 'x509' registered [ 3.550848] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.556794] io scheduler mq-deadline registered [ 3.561519] io scheduler kyber registered [ 3.567577] io scheduler bfq registered [ 3.571396] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.575644] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.580429] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.584111] ACPI: Power Button [PWRF] [ 3.741069] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.838407] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.057262] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.157654] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.436249] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.476554] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.514777] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.519508] Non-volatile memory driver v1.3 [ 4.520681] Linux agpgart interface v0.103 [ 4.555702] virtio_blk virtio1: [vda] 67992 512-byte logical blocks (34.8 MB/33.2 MiB) [ 4.558802] vda: detected capacity change from 0 to 34811904 [ 4.584493] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.587792] vdb: detected capacity change from 0 to 1073741824 [ 4.619676] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.624701] vdc: detected capacity change from 0 to 2621440000 [ 4.653699] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.657051] vdd: detected capacity change from 0 to 2621440000 [ 4.690949] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.699925] vde: detected capacity change from 0 to 4294967296 [ 4.731518] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.736362] vdf: detected capacity change from 0 to 4294967296 [ 4.747641] libphy: Fixed MDIO Bus: probed [ 4.760676] usbcore: registered new interface driver usbserial_generic [ 4.762482] usbserial: USB Serial support registered for generic [ 4.764428] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.771079] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.773097] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.775983] mousedev: PS/2 mouse device common for all mice [ 4.779274] rtc_cmos 00:05: RTC can wake from S4 [ 4.782260] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.785380] rtc_cmos 00:05: registered as rtc0 [ 4.791969] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.794068] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.803307] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.806204] intel_pstate: CPU model not supported [ 4.808952] hid: raw HID events driver (C) Jiri Kosina [ 4.810887] usbcore: registered new interface driver usbhid [ 4.813192] usbhid: USB HID core driver [ 4.814570] drop_monitor: Initializing network drop monitor service [ 4.817604] Initializing XFRM netlink socket [ 4.819559] NET: Registered protocol family 10 [ 4.824570] Segment Routing with IPv6 [ 4.826034] NET: Registered protocol family 17 [ 4.828363] mpls_gso: MPLS GSO support [ 4.842243] RAS: Correctable Errors collector initialized. [ 4.844046] AVX version of gcm_enc/dec engaged. [ 4.846832] AES CTR mode by8 optimization enabled [ 4.963746] sched_clock: Marking stable (4963090327, 0)->(6174236930, -1211146603) [ 4.970969] registered taskstats version 1 [ 4.972993] Loading compiled-in X.509 certificates [ 4.977569] zswap: loaded using pool lzo/zbud [ 5.008110] Key type big_key registered [ 5.037259] Key type encrypted registered [ 5.038780] ima: No TPM chip found, activating TPM-bypass! [ 5.041810] ima: Allocated hash algorithm: sha1 [ 5.045085] ima: No architecture policies found [ 5.046628] evm: Initialising EVM extended attributes: [ 5.048860] evm: security.selinux [ 5.050530] evm: security.ima [ 5.052151] evm: security.capability [ 5.053558] evm: HMAC attrs: 0x1 [ 5.058945] rtc_cmos 00:05: setting system clock to 2025-10-24 16:02:49 UTC (1761321769) [ 5.070472] debug: unmapping init [mem 0xffffffff91803000-0xffffffff919fffff] [ 5.073390] debug: unmapping init [mem 0xffffffff90582000-0xffffffff90858fff] [ 5.085111] Write protecting the kernel read-only data: 28672k [ 5.089875] debug: unmapping init [mem 0xffffffff8ec03000-0xffffffff8edfffff] [ 5.095377] debug: unmapping init [mem 0xffffffff8f514000-0xffffffff8f5fffff] [ 5.150259] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.161855] systemd[1]: Detected virtualization kvm. [ 5.165388] systemd[1]: Detected architecture x86-64. [ 5.166968] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.195311] systemd[1]: No hostname configured. [ 5.197873] systemd[1]: Set hostname to . [ 5.199891] random: systemd: uninitialized urandom read (16 bytes read) [ 5.202167] systemd[1]: Initializing machine ID from random generator. [ 5.296368] random: ln: uninitialized urandom read (6 bytes read) [ 5.523847] random: systemd: uninitialized urandom read (16 bytes read) [ 5.527276] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.535965] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.560809] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.652715] device-mapper: uevent: version 1.0.3 [ 6.654765] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ 7.182242] hrtimer: interrupt took 20004558 ns [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.304976] virtio_net virtio0 ens2: renamed from eth0 [ 8.522879] random: fast init done [ 8.633797] scsi host0: ata_piix [ 8.640538] scsi host1: ata_piix [ 8.674268] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.684393] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.586226] random: crng init done [ 13.587637] random: 7 urandom warning(s) missed due to ratelimiting [ 15.190400] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.962233] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.844189] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.249226] SELinux: Disabled at runtime. [ 20.326634] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.339619] systemd[1]: Detected virtualization kvm. [ 20.342801] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.377809] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.383169] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.390445] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.402750] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.407427] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.417982] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.427162] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ 21.569798] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 22.507673] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.262980] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.300412] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.758317] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.864358] EDAC sbridge: Ver: 1.1.2 [ 27.762821] Key type dns_resolver registered [* ] A start j[ 28.166743] NFS: Registering the id_resolver key type ob is running for Configur…-on[ 28.181393] Key type id_resolver registered ly root support [ 28.185441] Key type id_legacy registered (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg132-server login: [ 52.158601] libcfs: loading out-of-tree module taints kernel. [ 52.184368] alg: No test for adler32 (adler32-zlib) [ 52.936356] Key type ._llcrypt registered [ 52.937840] Key type .llcrypt registered [ 52.982484] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 60.921052] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 61.514486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.795217] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 62.117995] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 62.120985] LNet: Accept secure, port 988 [ 63.751163] Key type lgssc registered [ 64.325853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.142476] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 73.643294] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.200366] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.733977] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.810925] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 93.221496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.265382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 94.388207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 94.405176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 94.449443] Lustre: lustre-MDT0000: new disk, initializing [ 94.495623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 94.504554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 96.285878] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 102.562743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.605479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 102.609545] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 102.716191] Lustre: lustre-OST0000: new disk, initializing [ 102.718959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 102.721849] Lustre: Skipped 1 previous similar message [ 102.746912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 104.387076] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 110.578210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 110.618198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 110.621537] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 110.662283] Lustre: lustre-OST0001: new disk, initializing [ 110.664602] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 110.682169] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 112.302248] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 118.158294] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 123.007418] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 129.577615] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing check_logdir /tmp/testlogs/ [ 131.465842] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing yml_node [ 133.735371] Lustre: DEBUG MARKER: Client: 2.15.7.12 [ 134.949952] Lustre: DEBUG MARKER: MDS: 2.15.7.12 [ 136.135133] Lustre: DEBUG MARKER: OSS: 2.15.7.12 [ 136.830550] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 12:05:00 EDT 2025 [ 139.966424] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 140.525094] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 156.127804] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 156.128145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 156.133849] Lustre: Skipped 1 previous similar message [ 158.589079] Lustre: server umount lustre-MDT0000 complete [ 160.137863] LustreError: 5657:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321924 with bad export cookie 18253209475555167163 [ 160.138686] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 160.142773] LustreError: 5657:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 166.199401] Lustre: server umount lustre-OST0000 complete [ 182.239185] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 182.329665] Lustre: server umount lustre-OST0001 complete [ 185.359286] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 188.763115] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 193.217805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 195.975691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 198.954624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.131394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.168752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.270152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 202.288418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 202.329849] Lustre: lustre-MDT0000: new disk, initializing [ 202.371243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.382537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.294709] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 209.166062] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 211.936521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 211.965119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.969798] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 212.089821] Lustre: lustre-OST0000: new disk, initializing [ 212.092313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 212.095216] Lustre: Skipped 1 previous similar message [ 212.124654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.899872] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 218.447151] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 220.318091] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 220.406771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 223.712283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 223.712694] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 223.718202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 223.724618] Lustre: Skipped 1 previous similar message [ 227.336406] Lustre: server umount lustre-OST0000 complete [ 235.487168] Lustre: 14611:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761321993/real 1761321993] req@00000000dbf3df34 x1846879795044608/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761321999 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 235.582068] Lustre: server umount lustre-MDT0000 complete [ 237.622915] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:06:41 (1761322001) [ 238.935191] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 239.799917] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:06:43 (1761322003) [ 242.875073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 243.133541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.970680] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 248.021149] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 250.917903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.043303] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 256.457737] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 268.768595] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 268.768623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.779923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 270.717188] Lustre: server umount lustre-OST0000 complete [ 279.007377] Lustre: 17160:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322037/real 1761322037] req@0000000037a38e4a x1846879795050112/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322043 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 284.899644] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 286.232719] Key type lgssc unregistered [ 286.376522] LNet: 17763:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.380546] LNet: Removed LNI 192.168.201.132@tcp [ 286.815550] Key type .llcrypt unregistered [ 286.817839] Key type ._llcrypt unregistered [ 297.276817] alg: No test for adler32 (adler32-zlib) [ 298.028424] Key type ._llcrypt registered [ 298.030115] Key type .llcrypt registered [ 298.079506] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 298.506608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 298.707828] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 298.800693] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 298.803406] LNet: Accept secure, port 988 [ 300.415352] Key type lgssc registered [ 300.890394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 304.372968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 305.570471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 307.259246] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 310.552922] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 313.247689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 313.377799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 315.085725] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 318.351385] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 320.548770] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 330.720990] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 330.721091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.731192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 334.757927] Lustre: server umount lustre-OST0000 complete [ 342.495146] Lustre: 21103:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322100/real 1761322100] req@00000000a8f066b6 x1846880043537920/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322106 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 342.578637] Lustre: server umount lustre-MDT0000 complete [ 347.818986] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 348.999949] Key type lgssc unregistered [ 349.119355] LNet: 21706:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.123099] LNet: Removed LNI 192.168.201.132@tcp [ 349.449439] Key type .llcrypt unregistered [ 349.451025] Key type ._llcrypt unregistered [ 359.033691] alg: No test for adler32 (adler32-zlib) [ 359.785455] Key type ._llcrypt registered [ 359.787314] Key type .llcrypt registered [ 359.836282] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 360.249857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 360.497908] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 360.593793] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 360.596785] LNet: Accept secure, port 988 [ 362.191763] Key type lgssc registered [ 362.654635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 365.999676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.213414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.991071] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 372.188568] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 375.048899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 375.185363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.935380] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 380.161259] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 383.333301] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 391.789445] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.32@tcp (stopping) [ 393.697033] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 393.703158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 395.271843] Lustre: server umount lustre-MDT0000 complete [ 405.983139] Lustre: 22480:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322163/real 1761322163] req@00000000b02e71eb x1846880108548736/t0(0) o400->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322170 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 405.991812] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 413.551351] Lustre: server umount lustre-OST0000 complete [ 420.023893] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 421.184955] Key type lgssc unregistered [ 421.317428] LNet: 25826:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.321369] LNet: Removed LNI 192.168.201.132@tcp [ 421.693684] Key type .llcrypt unregistered [ 421.695523] Key type ._llcrypt unregistered [ 431.476461] alg: No test for adler32 (adler32-zlib) [ 432.228374] Key type ._llcrypt registered [ 432.229888] Key type .llcrypt registered [ 432.270236] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 432.632755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 432.861954] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 432.961948] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 432.964349] LNet: Accept secure, port 988 [ 434.559276] Key type lgssc registered [ 434.926547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 437.648666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.838234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.365054] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 443.109304] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 445.677886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 445.806302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.317461] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 450.129632] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 453.987494] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 459.232123] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 459.232160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.241709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 461.929185] Lustre: lustre-OST0000: Not available for connect from 192.168.201.32@tcp (stopping) [ 464.354332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.000765] Lustre: server umount lustre-OST0000 complete [ 490.975176] Lustre: 29337:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322249/real 1761322249] req@0000000008bef1e8 x1846880184047168/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322255 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 491.039042] Lustre: server umount lustre-MDT0000 complete [ 496.488585] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 497.603356] Key type lgssc unregistered [ 497.730466] LNet: 29942:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.735420] LNet: Removed LNI 192.168.201.132@tcp [ 498.043966] Key type .llcrypt unregistered [ 498.045769] Key type ._llcrypt unregistered [ 507.895330] alg: No test for adler32 (adler32-zlib) [ 508.647349] Key type ._llcrypt registered [ 508.649012] Key type .llcrypt registered [ 508.691053] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 509.061094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 509.251390] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 509.339828] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 509.342279] LNet: Accept secure, port 988 [ 510.927212] Key type lgssc registered [ 511.324095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 514.228484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.408869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.044593] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 520.047803] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 522.838761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 522.972843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 524.721360] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 527.826908] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 531.109509] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 536.252977] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 537.578314] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 543.716567] LNetError: 120-3: Refusing connection from 192.168.201.132 for 127.0.0.2@tcp: No matching NI [ 543.720255] LNetError: 30702:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 543.723777] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 548.407382] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-2412h11m52s [ 549.033782] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 551.904402] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.911511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 551.915888] LNetError: 120-3: Refusing connection from 192.168.201.132 for 127.0.0.2@tcp: No matching NI [ 551.919089] LNetError: 30699:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 551.922123] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 554.600220] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.32@tcp (stopping) [ 557.024747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 557.031491] LNetError: 120-3: Refusing connection from 192.168.201.132 for 127.0.0.2@tcp: No matching NI [ 557.036293] LNetError: 30700:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 557.039634] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 559.712458] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.32@tcp (stopping) [ 562.145559] LNetError: 120-3: Refusing connection from 192.168.201.132 for 127.0.0.2@tcp: No matching NI [ 562.149153] LNetError: 30701:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 562.152358] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 564.191187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 564.292454] Lustre: server umount lustre-MDT0000 complete [ 565.567322] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 568.181135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.255050] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 568.260861] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0x48f5246270806401 to 0x48f5246270806703 [ 568.265885] Lustre: MGC192.168.201.132@tcp: Connection restored to 192.168.201.132@tcp (at 0@lo) [ 568.413617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.454241] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 570.167237] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 571.479752] Lustre: DEBUG MARKER: Wait for df (24187) ... [ 572.066433] Lustre: DEBUG MARKER: done [ 573.407849] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 573.411469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.417498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.420041] Lustre: Skipped 1 previous similar message [ 575.202133] Lustre: server umount lustre-OST0000 complete [ 583.135216] Lustre: 34725:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322341/real 1761322341] req@00000000e1ef2b23 x1846880263744192/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322347 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 583.226599] Lustre: server umount lustre-MDT0000 complete [ 588.434540] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 589.721260] Key type lgssc unregistered [ 589.859540] LNet: 35327:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.864443] LNet: Removed LNI 192.168.201.132@tcp [ 590.237873] Key type .llcrypt unregistered [ 590.239230] Key type ._llcrypt unregistered [ 601.422773] alg: No test for adler32 (adler32-zlib) [ 602.175502] Key type ._llcrypt registered [ 602.177089] Key type .llcrypt registered [ 602.221521] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 608.772448] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 609.243838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 609.390893] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 609.475490] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 609.477443] LNet: Accept secure, port 988 [ 611.063104] Key type lgssc registered [ 611.459252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 615.228128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 617.692911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 620.133034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 622.736969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 622.760622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.872926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 623.886884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 623.928824] Lustre: lustre-MDT0000: new disk, initializing [ 623.959825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 623.969102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 625.658785] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 629.944491] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 632.655487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 632.686544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 632.691258] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 632.797640] Lustre: lustre-OST0000: new disk, initializing [ 632.800156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 632.803359] Lustre: Skipped 1 previous similar message [ 632.830656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.631301] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 639.160174] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 640.903802] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 641.004461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 643.553170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.553273] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 643.558743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 648.672588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 653.792700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 655.839207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 655.917364] Lustre: server umount lustre-OST0000 complete [ 663.519167] Lustre: 41130:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322421/real 1761322421] req@0000000049994146 x1846880368597248/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322427 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 663.592439] Lustre: server umount lustre-MDT0000 complete [ 666.870870] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 12:13:50 (1761322430) [ 669.539873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.737217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 671.423674] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 674.226391] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 677.015916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 677.160377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 678.998300] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 682.302701] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 691.110356] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-2412h14m15s [ 691.734950] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 694.541744] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 695.577260] Lustre: DEBUG MARKER: Creating a test file and stat it [ 695.596081] LustreError: 41799:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 699.876610] LNetError: 120-3: Refusing connection from 192.168.201.132 for 127.0.0.2@tcp: No matching NI [ 699.880138] LNetError: 37277:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 699.884295] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 715.884968] Lustre: lustre-MDT0000: Client 60c88490-a8f3-454f-bc21-d9fb71130efd (at 192.168.201.32@tcp) reconnecting [ 735.639152] LustreError: 41799:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 736.509406] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 737.457957] Lustre: DEBUG MARKER: done [ 740.833179] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 740.837347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 740.842712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 745.446353] Lustre: server umount lustre-OST0000 complete [ 753.119156] Lustre: 44442:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322511/real 1761322511] req@0000000083893815 x1846880368607488/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322517 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 753.186805] Lustre: server umount lustre-MDT0000 complete [ 758.365596] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 759.462208] Key type lgssc unregistered [ 759.586337] LNet: 45046:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.589800] LNet: Removed LNI 192.168.201.132@tcp [ 759.908071] Key type .llcrypt unregistered [ 759.909493] Key type ._llcrypt unregistered [ 770.044424] alg: No test for adler32 (adler32-zlib) [ 770.796334] Key type ._llcrypt registered [ 770.797736] Key type .llcrypt registered [ 770.840950] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 776.985643] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 777.478148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 777.620581] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 777.708116] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 777.710267] LNet: Accept secure, port 988 [ 779.295157] Key type lgssc registered [ 779.719835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 783.097623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 785.297527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 787.602941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 790.158761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 790.181924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.284509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 791.296649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 791.337707] Lustre: lustre-MDT0000: new disk, initializing [ 791.370997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.381260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 792.950141] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 797.133896] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 799.691638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 799.721815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 799.725682] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 799.826267] Lustre: lustre-OST0000: new disk, initializing [ 799.829189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 799.832376] Lustre: Skipped 1 previous similar message [ 799.860187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 801.593726] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 805.788148] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 807.397892] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 807.483877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 810.976252] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 810.976443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 810.985926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 814.306472] Lustre: server umount lustre-OST0000 complete [ 822.239179] Lustre: 50845:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322580/real 1761322580] req@0000000049b35270 x1846880545806208/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322586 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 822.316611] Lustre: server umount lustre-MDT0000 complete [ 825.326608] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 12:16:29 (1761322589) [ 825.869270] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 826.481699] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 12:16:30 (1761322590) [ 830.465540] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 832.043192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.213653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 832.625261] Lustre: Failing over lustre-MDT0000 [ 839.135325] Lustre: 52024:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322597/real 1761322597] req@0000000065961d98 x1846880545808256/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322603 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 839.205101] Lustre: server umount lustre-MDT0000 complete [ 841.784538] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:16:45 (1761322605) [ 844.425864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.624604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.356316] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 849.387578] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 851.915284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 852.043170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 853.748202] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 856.742952] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 865.269546] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 870.368034] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.374166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 873.280974] Lustre: server umount lustre-MDT0000 complete [ 874.548963] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 876.283591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 879.413203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 879.464538] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.470374] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0x2d783a0c80aeb2fe to 0x2d783a0c80aeb910 [ 879.475116] Lustre: MGC192.168.201.132@tcp: Connection restored to 192.168.201.132@tcp (at 0@lo) [ 879.594110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.107979] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 883.768143] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 885.727878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 890.847994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 891.764904] Lustre: server umount lustre-MDT0000 complete [ 894.153303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 897.920255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 897.993872] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 898.000127] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0x2d783a0c80aeb910 to 0x2d783a0c80aebd31 [ 898.005837] Lustre: MGC192.168.201.132@tcp: Connection restored to (at 0@lo) [ 898.161539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.191343] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 900.094961] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 903.012962] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 911.227813] Lustre: server umount lustre-MDT0000 complete [ 913.676764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 914.932278] Lustre: DEBUG MARKER: files compared the same [ 915.654263] LustreError: 52573:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322680 with bad export cookie 3276432554284989745 [ 915.656422] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 915.660053] LustreError: 52573:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 921.763147] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 922.902409] Key type lgssc unregistered [ 923.022374] LNet: 58975:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.026119] LNet: Removed LNI 192.168.201.132@tcp [ 923.332664] Key type .llcrypt unregistered [ 923.333943] Key type ._llcrypt unregistered [ 933.005891] alg: No test for adler32 (adler32-zlib) [ 933.758554] Key type ._llcrypt registered [ 933.760234] Key type .llcrypt registered [ 933.802768] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 934.161308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 934.381241] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 934.464958] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 934.467340] LNet: Accept secure, port 988 [ 936.055128] Key type lgssc registered [ 936.436942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 939.312413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.475989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.997828] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug +malloc all 8 [ 944.815444] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 947.236058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.334385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 948.931363] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug +malloc all 8 [ 951.664633] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 956.513497] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 962.015914] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 962.015979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.023295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 962.522203] Lustre: server umount lustre-OST0000 complete [ 970.207408] Lustre: 62315:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322728/real 1761322728] req@0000000013858b52 x1846880709383168/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322734 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 970.268834] Lustre: server umount lustre-MDT0000 complete [ 975.214615] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 976.302822] Key type lgssc unregistered [ 976.429370] LNet: 62918:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.432542] LNet: Removed LNI 192.168.201.132@tcp [ 976.748659] Key type .llcrypt unregistered [ 976.750489] Key type ._llcrypt unregistered [ 985.637833] alg: No test for adler32 (adler32-zlib) [ 986.389344] Key type ._llcrypt registered [ 986.390995] Key type .llcrypt registered [ 986.431825] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 986.861311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 986.993971] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 987.072211] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 987.074074] LNet: Accept secure, port 988 [ 988.655097] Key type lgssc registered [ 988.992745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 991.641534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1008.095341] LustreError: 64349:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.132@tcp: failed processing log, type 1: rc = -5 [ 1039.839334] LustreError: 64349:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.132@tcp: failed processing log, type 4: rc = -110 [ 1070.562857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.050204] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1085.277435] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1094.578352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1094.770147] Lustre: *** cfs_fail_loc=706, val=0*** [ 1116.866854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.939365] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1119.710756] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1124.627296] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1127.392962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.393941] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1127.403273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1132.515030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.637813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1140.191133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1140.343173] Lustre: server umount lustre-OST0000 complete [ 1149.407188] Lustre: 66325:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322907/real 1761322907] req@0000000040052ceb x1846880764958272/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322913 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1149.523568] Lustre: server umount lustre-MDT0000 complete [ 1155.664528] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1157.248295] Key type lgssc unregistered [ 1157.440970] LNet: 66928:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.445753] LNet: Removed LNI 192.168.201.132@tcp [ 1157.918633] Key type .llcrypt unregistered [ 1157.921741] Key type ._llcrypt unregistered [ 1172.674887] alg: No test for adler32 (adler32-zlib) [ 1173.433093] Key type ._llcrypt registered [ 1173.437439] Key type .llcrypt registered [ 1173.555153] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1174.137610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1174.526501] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1174.675443] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1174.678725] LNet: Accept secure, port 988 [ 1176.295133] Key type lgssc registered [ 1176.946403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1182.523059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.113267] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1190.837934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1191.084745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.817686] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1199.767248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.807783] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1202.414986] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1215.458826] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.462486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.473726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1216.484770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1218.395756] Lustre: server umount lustre-OST0000 complete [ 1221.204838] Lustre: server umount lustre-MDT0000 complete [ 1228.767238] Lustre: 70818:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761322987/real 1761322987] req@00000000656f53ba x1846880962090048/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761322993 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1228.801553] Lustre: server umount MGS complete [ 1234.951229] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:23:18 (1761322998) [ 1250.473928] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 1255.394387] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1261.467644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1265.653382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1269.440205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1273.707396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1273.762952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.468644] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1283.460748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1283.514394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1283.518344] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1283.679750] Lustre: lustre-OST0000: new disk, initializing [ 1283.684073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1283.742117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1286.549709] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1293.374791] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1297.770373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1297.799518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1297.851714] Lustre: lustre-MDT0000: new disk, initializing [ 1297.951186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.966189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1300.703945] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1323.489160] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1323.498812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.517452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1328.223808] Lustre: server umount lustre-OST0000 complete [ 1337.009340] Lustre: server umount lustre-MDT0000 complete [ 1345.503379] Lustre: 76799:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323103/real 1761323103] req@00000000c2ccefe2 x1846880962096960/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323109 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1345.544837] Lustre: server umount MGS complete [ 1350.598738] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:25:14 (1761323114) [ 1361.395702] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1363.024020] Key type lgssc unregistered [ 1363.252881] LNet: 77900:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.263521] LNet: Removed LNI 192.168.201.132@tcp [ 1363.757175] Key type .llcrypt unregistered [ 1363.759099] Key type ._llcrypt unregistered [ 1374.103774] alg: No test for adler32 (adler32-zlib) [ 1374.862478] Key type ._llcrypt registered [ 1374.864275] Key type .llcrypt registered [ 1374.944329] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1375.565989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1375.827382] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1375.951634] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1375.966124] LNet: Accept secure, port 988 [ 1377.610136] Key type lgssc registered [ 1378.213590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1383.206477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.501496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.942868] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1391.091806] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1395.261555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1395.447421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.204620] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1399.527819] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 1402.962257] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1410.015879] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1410.025591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.036685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1413.898857] Lustre: server umount lustre-OST0000 complete [ 1421.792360] Lustre: 81051:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323180/real 1761323180] req@000000000b415e70 x1846881172854144/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323186 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1421.950672] Lustre: server umount lustre-MDT0000 complete [ 1428.846800] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1430.580059] Key type lgssc unregistered [ 1430.777765] LNet: 81653:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.783403] LNet: Removed LNI 192.168.201.132@tcp [ 1431.338270] Key type .llcrypt unregistered [ 1431.340279] Key type ._llcrypt unregistered [ 1440.826833] alg: No test for adler32 (adler32-zlib) [ 1441.579465] Key type ._llcrypt registered [ 1441.581177] Key type .llcrypt registered [ 1441.660444] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1442.137191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1442.495763] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1442.605885] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1442.610894] LNet: Accept secure, port 988 [ 1444.247136] Key type lgssc registered [ 1444.849203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1448.480106] LustreError: 82780:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1448.483415] LustreError: 82779:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1448.487065] LustreError: 82780:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4997 [ 1448.490367] LustreError: 82780:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1448.498158] LustreError: 82780:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1448.503471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.780285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1452.196086] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing lsmod [ 1453.731844] LustreError: 83323:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 1453.736469] LustreError: 83322:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 1453.740364] LustreError: 83323:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4995 [ 1453.743798] LustreError: 83323:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1453.747605] LustreError: 83323:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1453.753380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.923846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1454.953646] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 1455.778470] Lustre: server umount lustre-MDT0000 complete [ 1457.959691] LustreError: 82799:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323222 with bad export cookie 8420413081909254833 [ 1457.967889] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1464.287180] Lustre: 83707:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323222/real 1761323222] req@00000000407dcd2c x1846881243106880/t0(0) o39->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761323228 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1464.414185] Lustre: server umount lustre-OST0000 complete [ 1469.801159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.123684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.777091] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1477.346335] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1481.541026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1481.771083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.107074] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1484.839201] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 1488.264794] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1493.986354] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1493.990441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.997901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1495.008133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1497.541131] Lustre: server umount lustre-OST0000 complete [ 1504.735219] Lustre: 86063:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323263/real 1761323263] req@00000000a7c3b806 x1846881243112640/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323269 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1504.841353] Lustre: server umount lustre-MDT0000 complete [ 1511.439767] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1513.214853] Key type lgssc unregistered [ 1513.396531] LNet: 86665:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.405784] LNet: Removed LNI 192.168.201.132@tcp [ 1513.963946] Key type .llcrypt unregistered [ 1513.966758] Key type ._llcrypt unregistered [ 1528.445339] alg: No test for adler32 (adler32-zlib) [ 1529.197428] Key type ._llcrypt registered [ 1529.199759] Key type .llcrypt registered [ 1529.277979] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1529.796169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1530.093854] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1530.218085] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1530.221724] LNet: Accept secure, port 988 [ 1531.831183] Key type lgssc registered [ 1532.522708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1537.147993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.418733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.755506] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1545.076191] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1548.981632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1549.170292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.636764] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1554.282113] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1556.113918] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1559.680269] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1561.209224] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 1564.641958] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1564.648383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.655077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.010845] Lustre: server umount lustre-OST0000 complete [ 1577.442629] Lustre: 90116:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323335/real 1761323335] req@000000003c5e38f1 x1846881334338304/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323341 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1577.568683] Lustre: server umount lustre-MDT0000 complete [ 1584.828659] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1586.688632] Key type lgssc unregistered [ 1586.893046] LNet: 90719:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.900390] LNet: Removed LNI 192.168.201.132@tcp [ 1587.475844] Key type .llcrypt unregistered [ 1587.477852] Key type ._llcrypt unregistered [ 1596.908323] alg: No test for adler32 (adler32-zlib) [ 1597.660588] Key type ._llcrypt registered [ 1597.661773] Key type .llcrypt registered [ 1597.747872] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1598.453142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1598.678343] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1598.819884] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1598.823389] LNet: Accept secure, port 988 [ 1600.463225] Key type lgssc registered [ 1601.142650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1606.052454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1607.342928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.920724] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1614.145479] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1618.174782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1618.375088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.774041] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1623.536187] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 1625.318076] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1631.204957] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1631.208586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.219018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1633.761128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1636.416926] Lustre: server umount lustre-OST0000 complete [ 1644.511333] Lustre: 93868:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323402/real 1761323402] req@000000005868e5c8 x1846881406686656/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323408 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1644.601122] Lustre: server umount lustre-MDT0000 complete [ 1651.621482] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1653.326046] Key type lgssc unregistered [ 1653.519634] LNet: 94472:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.526288] LNet: Removed LNI 192.168.201.132@tcp [ 1653.959300] Key type .llcrypt unregistered [ 1653.960639] Key type ._llcrypt unregistered [ 1666.622932] alg: No test for adler32 (adler32-zlib) [ 1667.376369] Key type ._llcrypt registered [ 1667.377993] Key type .llcrypt registered [ 1667.438768] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1667.988140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1668.192857] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1668.331757] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1668.336934] LNet: Accept secure, port 988 [ 1669.951659] Key type lgssc registered [ 1670.603570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1674.982161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1676.244038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1678.930808] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1683.425644] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1686.945056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1687.135984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.417834] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1691.245323] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 1694.083773] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1698.715434] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1698.823849] Lustre: lustre-MDT0000: root_squash is set to 0:0 [ 1700.781919] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1700.785091] Lustre: Skipped 1 previous similar message [ 1703.564838] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 1703.568219] Lustre: Skipped 1 previous similar message [ 1707.538207] Lustre: lustre-MDT0000: nosquash_nids is cleared [ 1707.561867] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1712.532223] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 1712.535767] Lustre: Skipped 1 previous similar message [ 1717.776793] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.201.32@tcp 192.168.0.[2,10]@tcp [ 1722.337546] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1722.337917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.350113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1726.425416] Lustre: server umount lustre-OST0000 complete [ 1734.623163] Lustre: 98957:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323493/real 1761323493] req@00000000788c0530 x1846881479046208/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323499 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1734.739992] Lustre: server umount lustre-MDT0000 complete [ 1741.747050] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1743.266833] Key type lgssc unregistered [ 1743.432473] LNet: 99560:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.436626] LNet: Removed LNI 192.168.201.132@tcp [ 1743.863979] Key type .llcrypt unregistered [ 1743.865873] Key type ._llcrypt unregistered [ 1759.120244] alg: No test for adler32 (adler32-zlib) [ 1759.875412] Key type ._llcrypt registered [ 1759.877156] Key type .llcrypt registered [ 1759.935281] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1760.450431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1760.679154] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1760.809775] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1760.812746] LNet: Accept secure, port 988 [ 1762.423256] Key type lgssc registered [ 1762.983163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1767.354170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.645503] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1768.649798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.689829] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.201.32@tcp 192.168.0.[2,10]@tcp [ 1770.967969] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1774.787812] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1778.038048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1778.194624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.316154] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1784.020711] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1784.360543] Lustre: lustre-OST0000: deleting orphan objects from 0x0:199 to 0x0:225 [ 1794.529196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.529212] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1794.537444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1799.651595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.054857] Lustre: server umount lustre-OST0000 complete [ 1808.351168] Lustre: 103099:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323566/real 1761323566] req@000000001a63ec22 x1846881576556416/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323572 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1808.429993] Lustre: server umount lustre-MDT0000 complete [ 1814.093319] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1815.448621] Key type lgssc unregistered [ 1815.631212] LNet: 103703:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.638911] LNet: Removed LNI 192.168.201.132@tcp [ 1816.121255] Key type .llcrypt unregistered [ 1816.123619] Key type ._llcrypt unregistered [ 1828.777853] alg: No test for adler32 (adler32-zlib) [ 1829.529407] Key type ._llcrypt registered [ 1829.531619] Key type .llcrypt registered [ 1829.593506] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 1836.245310] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1836.775466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1837.000335] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1837.136166] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1837.138974] LNet: Accept secure, port 988 [ 1838.767129] Key type lgssc registered [ 1839.462701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1845.080164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1848.805565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1852.392917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1855.695973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1855.728882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.842765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1856.857568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1856.900050] Lustre: lustre-MDT0000: new disk, initializing [ 1856.971097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.984531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1859.277811] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1865.076272] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1868.997458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1869.034697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1869.039397] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1869.182716] Lustre: lustre-OST0000: new disk, initializing [ 1869.187921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1869.192642] Lustre: Skipped 1 previous similar message [ 1869.235629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.674680] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1876.947196] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1879.431613] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1879.559754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1882.593847] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1882.600125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.607745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1886.742426] Lustre: server umount lustre-OST0000 complete [ 1895.391205] Lustre: 109132:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323653/real 1761323653] req@00000000db5b59c9 x1846881656248704/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323659 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1895.489245] Lustre: server umount lustre-MDT0000 complete [ 1899.373429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.628615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.070448] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1906.303464] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1910.029170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1910.216568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.550979] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1916.040325] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1918.337981] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1918.442529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1922.973384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1923.007183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1923.011234] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1923.083788] Lustre: lustre-OST0001: new disk, initializing [ 1923.087365] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1923.128052] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1925.406778] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 1930.364532] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 1930.498552] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1933.674140] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d90c2b9d000.ost_server_uuid 40 [ 1934.578871] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d90c2b9d000.ost_server_uuid in FULL state after 0 sec [ 1942.500773] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1942.505455] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.514129] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1944.269493] Lustre: server umount lustre-OST0001 complete [ 1947.616728] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.616947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.630961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1952.494790] Lustre: server umount lustre-OST0000 complete [ 1959.903933] Lustre: 113385:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323718/real 1761323718] req@000000008d5a863a x1846881656262144/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323724 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1960.013349] Lustre: server umount lustre-MDT0000 complete [ 1970.463544] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 1971.871112] Key type lgssc unregistered [ 1972.018329] LNet: 114293:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.024491] LNet: Removed LNI 192.168.201.132@tcp [ 1972.481731] Key type .llcrypt unregistered [ 1972.483552] Key type ._llcrypt unregistered [ 1986.124808] alg: No test for adler32 (adler32-zlib) [ 1986.876455] Key type ._llcrypt registered [ 1986.878089] Key type .llcrypt registered [ 1986.931225] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 1994.210967] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 1994.756357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1994.900110] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1994.995370] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 1994.998030] LNet: Accept secure, port 988 [ 1996.591219] Key type lgssc registered [ 1997.051716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2001.239565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2003.980288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2006.709117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2009.746498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2009.771788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.884617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2010.901529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2010.941081] Lustre: lustre-MDT0000: new disk, initializing [ 2010.988602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.997917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2012.927246] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2018.157604] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2021.208275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2021.236697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2021.240793] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2021.339623] Lustre: lustre-OST0000: new disk, initializing [ 2021.342766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2021.348103] Lustre: Skipped 1 previous similar message [ 2021.373622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.147938] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2027.525400] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2029.385833] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2029.472365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2034.144792] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2034.148557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.154638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2036.393736] Lustre: server umount lustre-OST0000 complete [ 2044.383280] Lustre: 120097:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323802/real 1761323802] req@0000000012367d54 x1846881821923456/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323808 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2044.459885] Lustre: server umount lustre-MDT0000 complete [ 2047.738972] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 12:36:51 (1761323811) [ 2053.282709] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 2056.726271] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2060.649607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2063.161139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2065.825460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2068.620449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2068.648952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.746996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2068.764879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2068.807822] Lustre: lustre-MDT0000: new disk, initializing [ 2068.844933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.853713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2070.625391] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2075.349281] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2078.212287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2078.241653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2078.244924] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2078.352737] Lustre: lustre-OST0000: new disk, initializing [ 2078.355945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2078.359869] Lustre: Skipped 1 previous similar message [ 2078.395873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.164954] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2085.211710] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2087.577903] Lustre: Failing over lustre-OST0000 [ 2087.672886] Lustre: server umount lustre-OST0000 complete [ 2096.992992] Lustre: 116253:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323854/real 1761323854] req@000000005f3304af x1846881821928960/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761323861 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2097.007452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.079351] Lustre: 116252:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323855/real 1761323855] req@00000000ac76f7fd x1846881821929152/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1761323862 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2102.079723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2102.216116] Lustre: 116255:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323859/real 1761323859] req@00000000f41fafc7 x1846881821929344/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761323866 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2102.221687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.236081] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2103.264937] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2104.072944] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2104.938135] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2104.938530] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2104.945667] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2106.042714] Lustre: Failing over lustre-MDT0000 [ 2106.229396] Lustre: server umount lustre-MDT0000 complete [ 2115.551172] Lustre: 116253:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323872/real 1761323872] req@00000000a5cbb574 x1846881821931776/t0(0) o400->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323879 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2115.568344] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2120.493236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.698984] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xe0fa40ded1b89da6 to 0xe0fa40ded1b8a188 [ 2121.704546] Lustre: MGC192.168.201.132@tcp: Connection restored to (at 0@lo) [ 2121.889576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.918353] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2122.354769] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2122.369328] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2122.390575] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2123.708876] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2127.331463] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.361462] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2129.921484] Lustre: server umount lustre-OST0000 complete [ 2138.079271] Lustre: 126793:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323896/real 1761323896] req@000000007e403b71 x1846881821935040/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761323902 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2144.290792] Lustre: server umount lustre-MDT0000 complete [ 2149.442764] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 2150.643380] Key type lgssc unregistered [ 2150.766464] LNet: 127396:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.770866] LNet: Removed LNI 192.168.201.132@tcp [ 2151.112147] Key type .llcrypt unregistered [ 2151.113875] Key type ._llcrypt unregistered [ 2161.465358] alg: No test for adler32 (adler32-zlib) [ 2162.217523] Key type ._llcrypt registered [ 2162.219364] Key type .llcrypt registered [ 2162.264245] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2162.691202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2162.889964] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2162.992048] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 2162.995169] LNet: Accept secure, port 988 [ 2164.591153] Key type lgssc registered [ 2165.015202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2168.371568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.577139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.237755] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2174.726593] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2177.842503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2178.018681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.015331] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2183.343794] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2185.192847] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 2301.555313] Lustre: lustre-MDT0000: Client 9f2e6ab9-6c21-47c5-95ed-afc06e382ec9 (at 192.168.201.32@tcp) reconnecting [ 2360.145976] Lustre: Failing over lustre-MDT0000 [ 2360.318687] Lustre: server umount lustre-MDT0000 complete [ 2371.423338] Lustre: 128168:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324128/real 1761324128] req@000000009b166730 x1846881998380864/t0(0) o400->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324135 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2371.433643] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.090436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.672263] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0x52653dcac45777df to 0x52653dcac459e2be [ 2376.677049] Lustre: MGC192.168.201.132@tcp: Connection restored to 192.168.201.132@tcp (at 0@lo) [ 2376.816407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.843865] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2378.498322] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2380.635536] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2380.653801] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2380.673464] Lustre: lustre-OST0000: deleting orphan objects from 0x0:100 to 0x0:129 [ 2382.306389] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.316092] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2383.086419] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2383.732690] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2397.664272] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.670595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2402.785518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2403.148460] Lustre: server umount lustre-MDT0000 complete [ 2404.792562] LustreError: 130631:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324169 with bad export cookie 5937219624844780222 [ 2404.795987] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2404.797641] LustreError: 130631:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2419.167389] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2419.273030] Lustre: server umount lustre-OST0000 complete [ 2423.150974] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 2426.630242] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2430.743892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2433.258731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2435.745814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2438.368578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2438.398793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.496600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2438.512453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2438.554806] Lustre: lustre-MDT0000: new disk, initializing [ 2438.590893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.599720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2440.250747] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2444.382357] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2447.142691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2447.173539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2447.177615] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2447.284760] Lustre: lustre-OST0000: new disk, initializing [ 2447.287542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2447.290084] Lustre: Skipped 1 previous similar message [ 2447.314795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.903672] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2452.931202] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2454.622879] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2454.702311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2460.128927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.129075] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2460.135589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2461.542624] Lustre: server umount lustre-OST0000 complete [ 2469.343157] Lustre: 136628:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324227/real 1761324227] req@0000000039573361 x1846881998392320/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324233 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2469.410957] Lustre: server umount lustre-MDT0000 complete [ 2472.526848] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:43:56 (1761324236) [ 2478.483284] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 2481.621193] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2485.022910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2487.209409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2489.457051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2491.940782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2491.965087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.055075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2492.068752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2492.109048] Lustre: lustre-MDT0000: new disk, initializing [ 2492.139180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.146777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2493.775533] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2497.732035] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2500.182060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2500.211711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.214510] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2500.311892] Lustre: lustre-OST0000: new disk, initializing [ 2500.314498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2500.317688] Lustre: Skipped 1 previous similar message [ 2501.907590] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2506.018462] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2516.448946] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2516.449058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.457890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.459340] Lustre: server umount lustre-OST0000 complete [ 2527.199220] Lustre: 141428:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324285/real 1761324285] req@00000000bfefbaa1 x1846881998398976/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324291 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2530.559448] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:44:54 (1761324294) [ 2535.411367] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 2538.085656] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2541.595883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2543.542151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2545.696104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2548.209901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2548.231321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.323582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2548.335906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2548.370781] Lustre: lustre-MDT0000: new disk, initializing [ 2548.395591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.398784] Lustre: Skipped 1 previous similar message [ 2548.404881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2550.005219] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2554.041479] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2556.457756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2556.486068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2556.489104] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2556.581246] Lustre: lustre-OST0000: new disk, initializing [ 2556.584242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2556.587320] Lustre: Skipped 1 previous similar message [ 2558.091063] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2562.038156] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2568.160294] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2568.162307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.164497] LustreError: Skipped 1 previous similar message [ 2568.170922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2570.143040] Lustre: server umount lustre-OST0000 complete [ 2570.145391] Lustre: Skipped 1 previous similar message [ 2576.863179] Lustre: 146233:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324335/real 1761324335] req@000000006f4d223f x1846881998405056/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324341 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2581.862328] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 2582.912290] Key type lgssc unregistered [ 2583.037446] LNet: 146835:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2583.041594] LNet: Removed LNI 192.168.201.132@tcp [ 2583.337141] Key type .llcrypt unregistered [ 2583.338386] Key type ._llcrypt unregistered [ 2592.881395] alg: No test for adler32 (adler32-zlib) [ 2593.633389] Key type ._llcrypt registered [ 2593.634397] Key type .llcrypt registered [ 2593.675333] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2594.049893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2594.271878] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2594.355556] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 2594.357723] LNet: Accept secure, port 988 [ 2595.943235] Key type lgssc registered [ 2596.313475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2599.145058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.352126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.857932] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2604.615771] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2606.986292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2607.089793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.409985] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2610.878274] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2615.268765] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2630.624783] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2630.625192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2630.632339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2635.170381] Lustre: server umount lustre-OST0000 complete [ 2642.911167] Lustre: 150198:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324401/real 1761324401] req@00000000b21a533c x1846882450019968/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324407 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2642.975326] Lustre: server umount lustre-MDT0000 complete [ 2647.724739] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 2648.770772] Key type lgssc unregistered [ 2648.881354] LNet: 150801:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2648.885426] LNet: Removed LNI 192.168.201.132@tcp [ 2649.192210] Key type .llcrypt unregistered [ 2649.193677] Key type ._llcrypt unregistered [ 2657.845445] alg: No test for adler32 (adler32-zlib) [ 2658.598258] Key type ._llcrypt registered [ 2658.599912] Key type .llcrypt registered [ 2658.639364] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2659.026156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2659.275704] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2659.365686] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 2659.368575] LNet: Accept secure, port 988 [ 2660.951125] Key type lgssc registered [ 2661.311573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2664.002704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.163366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.632608] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2669.057225] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2671.269593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2671.373309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2672.689388] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2675.161878] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2680.803764] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 2691.040544] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2691.044461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.049966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2691.690624] Lustre: lustre-OST0000: Not available for connect from 192.168.201.32@tcp (stopping) [ 2695.080235] Lustre: server umount lustre-OST0000 complete [ 2697.888521] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d90d86ec000.ost_server_uuid 40 [ 2698.454794] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2721.247150] Lustre: 154406:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324479/real 1761324479] req@00000000ece8a8af x1846882518177472/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324485 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2721.311836] Lustre: server umount lustre-MDT0000 complete [ 2724.194989] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:48:08 (1761324488) [ 2726.643856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.824288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2728.335919] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2730.902480] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2733.139826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2733.249172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.808502] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2737.417676] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2739.791295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2739.817146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2739.821437] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2739.877107] Lustre: lustre-OST0001: new disk, initializing [ 2739.879956] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2739.903434] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2741.499995] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2742.819885] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 2745.325537] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2747.873141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.873349] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2747.879395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2750.565784] Lustre: lustre-OST0000: Not available for connect from 192.168.201.32@tcp (stopping) [ 2752.307521] Lustre: server umount lustre-OST0000 complete [ 2752.992159] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2754.537407] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2754.639984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2755.679688] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.112096] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.799460] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.327812] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2767.331337] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.337584] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2768.353382] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.359436] LustreError: Skipped 1 previous similar message [ 2770.713703] Lustre: server umount lustre-OST0001 complete [ 2778.079221] Lustre: 158496:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324536/real 1761324536] req@00000000626d8aa3 x1846882518188480/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324542 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2778.150960] Lustre: server umount lustre-MDT0000 complete [ 2780.073138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2781.405960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2782.787325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2784.946237] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:49:09 (1761324549) [ 2787.283681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2787.377637] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2787.388684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2787.460693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2788.950502] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2791.454949] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2793.705833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2793.767052] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2795.066053] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2797.512333] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2799.726871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2799.753336] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2801.211283] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2801.855557] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:129 [ 2801.858703] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 2803.527808] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2807.265052] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2807.267557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.271542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2807.273636] Lustre: Skipped 1 previous similar message [ 2810.479353] Lustre: server umount lustre-OST0000 complete [ 2812.385836] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.624981] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2822.628720] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2827.311166] Lustre: server umount lustre-OST0001 complete [ 2833.887258] Lustre: 162850:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324592/real 1761324592] req@000000000809e427 x1846882518198592/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324598 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2835.974501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2837.370786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.700038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2840.856534] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:50:05 (1761324605) [ 2845.430362] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 2848.148590] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 2851.664448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2853.688015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2855.607178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2857.767115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2857.791298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.871239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2857.882787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2857.917408] Lustre: lustre-MDT0000: new disk, initializing [ 2857.939784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.942106] Lustre: Skipped 2 previous similar messages [ 2857.948033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2859.345799] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2863.289202] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2865.569290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2865.592509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2865.596609] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2865.683744] Lustre: lustre-OST0000: new disk, initializing [ 2865.685485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2865.687026] Lustre: Skipped 1 previous similar message [ 2867.145521] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2870.914832] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2872.398773] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2872.476357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2876.384650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.384705] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2876.388488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2876.394013] Lustre: Skipped 2 previous similar messages [ 2879.258840] Lustre: server umount lustre-OST0000 complete [ 2879.260299] Lustre: Skipped 1 previous similar message [ 2885.599139] Lustre: 168487:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324644/real 1761324644] req@00000000f12d6e2c x1846882518204480/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324650 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2888.506917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.980683] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2892.444610] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2894.737413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2894.834049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.836164] Lustre: Skipped 2 previous similar messages [ 2896.162987] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2898.553774] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2900.034717] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2904.199806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2909.152147] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2909.152606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.161218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2912.997134] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2913.076934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2936.622595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2938.195665] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2940.768096] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2940.834957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2940.839038] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 2940.842643] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 2945.365558] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 2951.136472] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2951.136746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2952.354569] Lustre: server umount lustre-OST0000 complete [ 2952.356744] Lustre: Skipped 2 previous similar messages [ 2959.839174] Lustre: 172737:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324718/real 1761324718] req@000000003e00e4fb x1846882518213056/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324724 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2962.806488] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:52:06 (1761324726) [ 2964.880269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.028991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.031817] Lustre: Skipped 1 previous similar message [ 2966.407178] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2968.982652] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2971.335046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2972.917204] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2975.466944] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2976.955090] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2981.142351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2983.519487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2983.541835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2983.544913] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2983.586215] Lustre: lustre-OST0001: new disk, initializing [ 2983.587909] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2984.953436] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 2988.823595] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2990.365069] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2991.481316] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2995.682326] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.686898] LustreError: Skipped 4 previous similar messages [ 2996.407179] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2996.490363] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3020.089399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.477702] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3023.944240] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3024.502756] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3025.376051] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3025.377572] Lustre: Skipped 1 previous similar message [ 3035.618574] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.623554] LustreError: Skipped 10 previous similar messages [ 3045.857048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.863358] Lustre: Skipped 1 previous similar message [ 3065.311125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3071.455160] Lustre: 178735:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324829/real 1761324829] req@000000000002633b x1846882518228608/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324835 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3073.532722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3074.823052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3076.101040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3078.278470] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:54:02 (1761324842) [ 3080.589796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.709178] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3080.719551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3082.327965] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3085.245566] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3087.763232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3087.836666] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3089.433760] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3092.160561] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3096.540099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.581332] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3096.647403] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3096.651037] Lustre: Skipped 5 previous similar messages [ 3098.181180] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3101.059805] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3101.760828] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3103.424857] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3103.510593] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3105.597962] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d90c423c000.ost_server_uuid 40 [ 3106.123257] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d90c423c000.ost_server_uuid in FULL state after 0 sec [ 3106.442628] Lustre: Permanently deactivating lustre-OST0001 [ 3106.447078] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3107.211719] Lustre: Permanently reactivating lustre-OST0001 [ 3108.321172] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3108.324543] LustreError: Skipped 1 previous similar message [ 3114.228418] Lustre: server umount lustre-OST0001 complete [ 3114.230056] Lustre: Skipped 5 previous similar messages [ 3118.560419] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.565863] LustreError: Skipped 2 previous similar messages [ 3129.311130] Lustre: 183807:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324887/real 1761324887] req@00000000120712d2 x1846882518243072/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324893 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3131.336785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3132.627167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3133.853333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3135.873963] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:55:00 (1761324900) [ 3136.220716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3138.407996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.498455] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3138.508721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3138.511274] Lustre: Skipped 3 previous similar messages [ 3140.004349] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3142.451361] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3144.702964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3144.730826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3144.799756] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3144.807986] Lustre: Permanently deactivating lustre-OST0000 [ 3146.335200] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3148.948216] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3151.244090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3152.625370] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3154.103961] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3154.106904] LustreError: 187590:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 3154.111461] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3154.114391] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 3155.149362] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3155.792839] Lustre: Permanently reactivating lustre-OST0000 [ 3155.797782] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 3155.800127] Lustre: Skipped 2 previous similar messages [ 3163.124838] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3163.127048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3163.132084] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 3163.133104] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 3168.226994] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3168.229891] Lustre: Skipped 7 previous similar messages [ 3185.631211] Lustre: 188276:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324944/real 1761324944] req@00000000ee5db71b x1846882518256640/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324950 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3190.388091] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 3191.369737] Key type lgssc unregistered [ 3191.483377] LNet: 188878:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3191.485846] LNet: Removed LNI 192.168.201.132@tcp [ 3191.745878] Key type .llcrypt unregistered [ 3191.747568] Key type ._llcrypt unregistered [ 3200.491216] alg: No test for adler32 (adler32-zlib) [ 3201.242399] Key type ._llcrypt registered [ 3201.243594] Key type .llcrypt registered [ 3201.279747] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 3206.639877] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3206.913697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3207.124795] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3207.199115] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 3207.201290] LNet: Accept secure, port 988 [ 3208.775124] Key type lgssc registered [ 3209.085542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3211.953685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.955548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3216.055521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3218.317550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3218.340149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.433440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3219.446515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3219.483687] Lustre: lustre-MDT0000: new disk, initializing [ 3219.508511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.516758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3220.926125] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3224.817375] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3227.081875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3227.104159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3227.108532] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3227.190516] Lustre: lustre-OST0000: new disk, initializing [ 3227.193410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3227.197207] Lustre: Skipped 1 previous similar message [ 3227.219302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.535064] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3232.513775] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3239.506817] LustreError: 193755:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3243.669813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3243.690224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3243.692451] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3243.733598] Lustre: lustre-OST0001: new disk, initializing [ 3243.736342] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3243.756744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3245.148747] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3248.999211] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3249.538401] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3249.599095] LustreError: 193755:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3249.606911] LustreError: 193755:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3259.695118] LustreError: 193755:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake [ 3261.920329] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3261.921017] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3261.927444] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3264.104306] Lustre: lustre-OST0001: Not available for connect from 192.168.201.32@tcp (stopping) [ 3266.611228] Lustre: server umount lustre-OST0001 complete [ 3267.041173] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.161900] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3277.280561] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.839814] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3279.842352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.847556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3282.400131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3282.400600] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.519952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3287.520434] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.151160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3293.219562] Lustre: server umount lustre-OST0000 complete [ 3306.975165] Lustre: 195399:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325065/real 1761325065] req@00000000e2eacf67 x1846883092805888/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325071 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3307.036151] Lustre: server umount lustre-MDT0000 complete [ 3311.894183] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 3312.895131] Key type lgssc unregistered [ 3313.021423] LNet: 196004:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3313.025790] LNet: Removed LNI 192.168.201.132@tcp [ 3313.337976] Key type .llcrypt unregistered [ 3313.339719] Key type ._llcrypt unregistered [ 3322.306534] alg: No test for adler32 (adler32-zlib) [ 3323.057385] Key type ._llcrypt registered [ 3323.058871] Key type .llcrypt registered [ 3323.093747] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 3328.980544] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3329.415109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3329.540647] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3329.619395] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 3329.621857] LNet: Accept secure, port 988 [ 3331.207137] Key type lgssc registered [ 3331.509893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3334.427723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3336.411582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3338.315767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3340.508242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3340.530823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.615623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3341.624327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3341.655932] Lustre: lustre-MDT0000: new disk, initializing [ 3341.676273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.681819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3343.050958] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3346.877366] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3349.174333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3349.197382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3349.199555] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3349.283563] Lustre: lustre-OST0000: new disk, initializing [ 3349.285563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3349.287305] Lustre: Skipped 1 previous similar message [ 3349.308947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3350.760268] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3354.583677] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3356.067166] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3356.136907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3361.248408] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3361.248681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.256389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3362.904106] Lustre: server umount lustre-OST0000 complete [ 3370.463231] Lustre: 201800:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325128/real 1761325128] req@00000000b71cc013 x1846883221772160/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325134 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3370.552826] Lustre: server umount lustre-MDT0000 complete [ 3373.566582] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 12:58:57 (1761325137) [ 3375.917272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.109096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.610711] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3380.203099] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3382.623570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3382.739573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.277243] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3387.024199] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3401.184190] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3401.184380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.192758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3404.705817] Lustre: server umount lustre-OST0000 complete [ 3406.120289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3410.010531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3410.107984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.420996] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3414.031406] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3417.311883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.625173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3423.068552] Lustre: server umount lustre-OST0000 complete [ 3429.855246] Lustre: 206106:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325188/real 1761325188] req@000000002bb8e311 x1846883221780032/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325194 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3429.923647] Lustre: server umount lustre-MDT0000 complete [ 3434.929518] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 3436.023928] Key type lgssc unregistered [ 3436.146233] LNet: 206708:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3436.150489] LNet: Removed LNI 192.168.201.132@tcp [ 3436.462264] Key type .llcrypt unregistered [ 3436.463967] Key type ._llcrypt unregistered [ 3445.526901] alg: No test for adler32 (adler32-zlib) [ 3446.278470] Key type ._llcrypt registered [ 3446.279435] Key type .llcrypt registered [ 3446.321087] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3446.881488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3447.045738] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3447.134048] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 3447.135833] LNet: Accept secure, port 988 [ 3448.719146] Key type lgssc registered [ 3449.093312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3451.930187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.105864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.632324] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3457.097060] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3459.365952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3459.472888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3460.753106] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3463.127967] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3467.621607] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3477.984326] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3477.984550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3477.993483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3482.834473] Lustre: server umount lustre-OST0000 complete [ 3490.271130] Lustre: 210822:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325248/real 1761325248] req@0000000026020874 x1846883344455168/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325254 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3490.347716] Lustre: server umount lustre-MDT0000 complete [ 3495.113057] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 3496.150439] Key type lgssc unregistered [ 3496.262484] LNet: 211425:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3496.266377] LNet: Removed LNI 192.168.201.132@tcp [ 3496.538241] Key type .llcrypt unregistered [ 3496.539868] Key type ._llcrypt unregistered [ 3503.256761] alg: No test for adler32 (adler32-zlib) [ 3504.008525] Key type ._llcrypt registered [ 3504.010275] Key type .llcrypt registered [ 3504.051253] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3504.507477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3504.769135] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3504.846611] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 3504.848941] LNet: Accept secure, port 988 [ 3506.431105] Key type lgssc registered [ 3506.763240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3509.359219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.543068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.979044] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3514.269464] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3516.502648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3516.625222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.005833] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3520.489298] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3525.796106] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 3536.352449] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3536.352791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.360632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3541.471896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3541.670430] Lustre: server umount lustre-OST0000 complete [ 3548.639083] Lustre: 214774:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325307/real 1761325307] req@00000000b69c73fb x1846883405272576/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325313 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3548.724798] Lustre: server umount lustre-MDT0000 complete [ 3553.413384] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 3554.388730] Key type lgssc unregistered [ 3554.499371] LNet: 215377:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3554.503328] LNet: Removed LNI 192.168.201.132@tcp [ 3554.821411] Key type .llcrypt unregistered [ 3554.823053] Key type ._llcrypt unregistered [ 3561.646201] alg: No test for adler32 (adler32-zlib) [ 3562.397347] Key type ._llcrypt registered [ 3562.398440] Key type .llcrypt registered [ 3562.439260] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3562.886045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3563.035297] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3563.125960] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 3563.127772] LNet: Accept secure, port 988 [ 3564.711090] Key type lgssc registered [ 3565.053780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3567.845459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3569.027834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.570605] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3573.090453] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3575.478713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3575.573775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3577.005870] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3579.718485] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3583.716250] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 3589.088477] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3589.088666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.095282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3590.564238] Lustre: server umount lustre-OST0000 complete [ 3598.303123] Lustre: 218522:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325356/real 1761325356] req@000000004e05c987 x1846883466089280/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325362 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3598.380132] Lustre: server umount lustre-MDT0000 complete [ 3603.319968] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 3604.348295] Key type lgssc unregistered [ 3604.464376] LNet: 219126:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3604.468320] LNet: Removed LNI 192.168.201.132@tcp [ 3604.821507] Key type .llcrypt unregistered [ 3604.823711] Key type ._llcrypt unregistered [ 3613.745933] alg: No test for adler32 (adler32-zlib) [ 3614.497353] Key type ._llcrypt registered [ 3614.499089] Key type .llcrypt registered [ 3614.540020] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3614.955791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3615.105145] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3615.184508] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 3615.187479] LNet: Accept secure, port 988 [ 3616.767159] Key type lgssc registered [ 3617.165097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3619.939995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.114308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.639753] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3625.235847] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3627.522398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.643678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.121980] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3631.617741] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3635.811956] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 3646.432702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.433058] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3646.439917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3651.551979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3651.681479] Lustre: server umount lustre-OST0000 complete [ 3659.231150] Lustre: 223288:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325417/real 1761325417] req@00000000cb198343 x1846883520615936/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325423 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3659.289610] Lustre: server umount lustre-MDT0000 complete [ 3664.058593] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 3664.924050] Key type lgssc unregistered [ 3665.029279] LNet: 223890:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3665.032683] LNet: Removed LNI 192.168.201.132@tcp [ 3665.305788] Key type .llcrypt unregistered [ 3665.307784] Key type ._llcrypt unregistered [ 3671.844060] alg: No test for adler32 (adler32-zlib) [ 3672.595327] Key type ._llcrypt registered [ 3672.596310] Key type .llcrypt registered [ 3672.627441] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3672.995674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3673.135773] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3673.202861] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 3673.204301] LNet: Accept secure, port 988 [ 3674.775093] Key type lgssc registered [ 3675.113844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3677.564574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.722977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.023070] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3682.622706] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3684.901398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3685.007091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.463222] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3688.799689] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3694.179621] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 3704.801091] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3704.801618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.811305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3705.969896] Lustre: server umount lustre-OST0000 complete [ 3713.503160] Lustre: 227240:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325471/real 1761325471] req@0000000083899c5f x1846883581433216/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325477 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3713.577774] Lustre: server umount lustre-MDT0000 complete [ 3718.640827] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 3719.814485] Key type lgssc unregistered [ 3719.954456] LNet: 227842:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3719.956966] LNet: Removed LNI 192.168.201.132@tcp [ 3720.285067] Key type .llcrypt unregistered [ 3720.286477] Key type ._llcrypt unregistered [ 3726.953848] alg: No test for adler32 (adler32-zlib) [ 3727.705341] Key type ._llcrypt registered [ 3727.707020] Key type .llcrypt registered [ 3727.742363] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3728.006884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3728.203825] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3728.267248] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 3728.268786] LNet: Accept secure, port 988 [ 3729.847165] Key type lgssc registered [ 3730.226291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3733.137304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3734.309046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.821460] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3738.429775] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3740.743450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3740.842224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3742.152568] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3744.601520] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3749.989974] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 3760.608427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3760.608638] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3760.612814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.707940] Lustre: server umount lustre-OST0000 complete [ 3772.383144] Lustre: 231008:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325530/real 1761325530] req@0000000006ce0d0c x1846883639104832/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325536 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3772.450113] Lustre: server umount lustre-MDT0000 complete [ 3777.069136] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 3777.990084] Key type lgssc unregistered [ 3778.097314] LNet: 231611:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3778.099988] LNet: Removed LNI 192.168.201.132@tcp [ 3778.384686] Key type .llcrypt unregistered [ 3778.386439] Key type ._llcrypt unregistered [ 3782.623624] alg: No test for adler32 (adler32-zlib) [ 3783.375360] Key type ._llcrypt registered [ 3783.376317] Key type .llcrypt registered [ 3783.408763] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3788.216796] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 3793.177874] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3793.564614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3793.778557] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3793.849413] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 3793.851734] LNet: Accept secure, port 988 [ 3795.431115] Key type lgssc registered [ 3795.793352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3798.683498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3800.682323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3802.829626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3805.129621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3805.155244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.249157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3806.259462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3806.300861] Lustre: lustre-MDT0000: new disk, initializing [ 3806.327562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.335430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3807.838324] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3811.714547] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3813.953356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3813.985106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3813.989400] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3814.091479] Lustre: lustre-OST0000: new disk, initializing [ 3814.093341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3814.095268] Lustre: Skipped 1 previous similar message [ 3814.115923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3815.652519] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3819.587500] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3821.123736] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3821.222506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3824.096839] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3824.097423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.104731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3828.082610] Lustre: server umount lustre-OST0000 complete [ 3834.847170] Lustre: 237270:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325593/real 1761325593] req@00000000c0e6d4e7 x1846883708311552/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325599 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3834.911541] Lustre: server umount lustre-MDT0000 complete [ 3837.677165] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:06:41 (1761325601) [ 3839.998246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.182071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.602188] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3844.153152] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3846.537308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3846.672064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3848.224658] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3850.791966] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3860.960388] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3860.960696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.969477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3865.702163] Lustre: server umount lustre-OST0000 complete [ 3873.247149] Lustre: 239825:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325631/real 1761325631] req@00000000602d32f4 x1846883708317056/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325637 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3873.317170] Lustre: server umount lustre-MDT0000 complete [ 3877.920549] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 3878.900745] Key type lgssc unregistered [ 3879.015287] LNet: 240428:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3879.017695] LNet: Removed LNI 192.168.201.132@tcp [ 3879.291527] Key type .llcrypt unregistered [ 3879.292802] Key type ._llcrypt unregistered [ 3887.968239] alg: No test for adler32 (adler32-zlib) [ 3888.720306] Key type ._llcrypt registered [ 3888.722067] Key type .llcrypt registered [ 3888.757152] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 3894.215872] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3894.552766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3894.755134] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3894.844087] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 3894.845792] LNet: Accept secure, port 988 [ 3896.431110] Key type lgssc registered [ 3896.744795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3899.638880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3901.639256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3903.779499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3905.580035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3907.413652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3909.860719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3909.887330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.988065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3910.998989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3911.038816] Lustre: lustre-MDT0000: new disk, initializing [ 3911.066762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3911.073640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3912.597750] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3916.518094] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3918.673204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3918.694631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3918.698262] Lustre: lustre-OST2710-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3918.775791] Lustre: lustre-OST2710: new disk, initializing [ 3918.778093] Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 [ 3918.780764] Lustre: Skipped 1 previous similar message [ 3918.801044] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 3920.211803] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3923.807750] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 3926.042291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3926.067493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3926.069627] Lustre: lustre-OST03e8-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3926.107690] Lustre: lustre-OST03e8: new disk, initializing [ 3926.109453] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 [ 3926.130051] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 3927.464594] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3929.980792] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 3932.633672] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 40 [ 3932.700607] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3933.960542] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 40 [ 3934.023529] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3937.248125] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.248648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3937.252490] Lustre: Skipped 1 previous similar message [ 3941.104303] Lustre: server umount lustre-MDT0000 complete [ 3942.311875] LustreError: 245008:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325706 with bad export cookie 4210256378362856931 [ 3942.313634] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3942.315711] LustreError: 245008:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3942.333714] Lustre: server umount lustre-OST2710 complete [ 3943.543162] Lustre: server umount lustre-OST03e8 complete [ 3948.613599] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 3951.561580] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 3954.788500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3956.867715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3959.007207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3961.195153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3961.218410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3961.296758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3961.307271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3961.348417] Lustre: lustre-MDT0000: new disk, initializing [ 3961.373302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3961.380356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3962.725703] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3966.453897] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3968.643799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3968.667293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3968.669756] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3968.763695] Lustre: lustre-OST0000: new disk, initializing [ 3968.765870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3968.767800] Lustre: Skipped 1 previous similar message [ 3968.789796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3970.306640] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 3974.227081] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3975.841807] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3976.943755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3979.674021] Lustre: server umount lustre-OST0000 complete [ 3987.423122] Lustre: 252586:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325745/real 1761325745] req@000000004371e7fc x1846883814228224/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761325751 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3993.567178] Lustre: 252586:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325751/real 1761325751] req@000000009903819e x1846883814228672/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325757 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3993.637913] Lustre: server umount lustre-MDT0000 complete [ 3996.659686] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:09:20 (1761325760) [ 3997.243603] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 3997.871317] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:09:22 (1761325762) [ 3998.971978] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4001.512251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4002.691938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4004.012599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4004.374805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4006.537564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.612882] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4006.620391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4006.676420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4008.137604] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4010.807630] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4013.330249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4013.354243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4013.428356] Lustre: Denying initial registration attempt from nid 192.168.201.132@tcp, specified as failover [ 4013.432898] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4013.436585] LustreError: 255539:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -99 [ 4013.439094] LustreError: 255539:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 4013.440913] LustreError: 255539:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 4013.493448] Lustre: server umount lustre-OST0000 complete [ 4013.494845] LustreError: 255539:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4020.703166] Lustre: 255809:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325779/real 1761325779] req@00000000b6e087b9 x1846883814231040/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325785 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4025.953248] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 4027.027427] Key type lgssc unregistered [ 4027.148470] LNet: 256414:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4027.151203] LNet: Removed LNI 192.168.201.132@tcp [ 4027.428347] Key type .llcrypt unregistered [ 4027.429628] Key type ._llcrypt unregistered [ 4031.714669] alg: No test for adler32 (adler32-zlib) [ 4032.466368] Key type ._llcrypt registered [ 4032.467451] Key type .llcrypt registered [ 4032.502468] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4032.832796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4033.052968] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4033.136820] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 4033.138272] LNet: Accept secure, port 988 [ 4034.719099] Key type lgssc registered [ 4035.019058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4037.279454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4038.489052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4039.725198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4040.074221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4045.857844] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4048.579396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.674150] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4049.683358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4049.737751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4051.129494] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4053.657748] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4055.975657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4056.006883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4056.075849] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4056.126380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.600724] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4059.936463] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4062.688176] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4062.691529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.697395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4066.727094] Lustre: server umount lustre-OST0000 complete [ 4074.463133] Lustre: 260998:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325832/real 1761325832] req@000000002f96198d x1846883958920960/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325838 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4074.529260] Lustre: server umount lustre-MDT0000 complete [ 4078.651682] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 4079.734955] Key type lgssc unregistered [ 4079.866391] LNet: 261601:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4079.869198] LNet: Removed LNI 192.168.201.132@tcp [ 4080.173154] Key type .llcrypt unregistered [ 4080.174603] Key type ._llcrypt unregistered [ 4089.722520] alg: No test for adler32 (adler32-zlib) [ 4090.474421] Key type ._llcrypt registered [ 4090.476969] Key type .llcrypt registered [ 4090.512206] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4090.949209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4091.089818] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4091.161955] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 4091.163610] LNet: Accept secure, port 988 [ 4092.743131] Key type lgssc registered [ 4093.127530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4095.867153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4097.015744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.343134] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4100.927401] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4103.185790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4103.292764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.720120] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4107.230678] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4117.472073] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4117.476817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4118.629266] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.32@tcp (stopping) [ 4121.867796] Lustre: server umount lustre-MDT0000 complete [ 4124.287717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4127.287678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.332679] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4127.336682] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0xeaf8130d55346926 to 0xeaf8130d553483ba [ 4127.341782] Lustre: MGC192.168.201.132@tcp: Connection restored to 192.168.201.132@tcp (at 0@lo) [ 4127.457075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.481789] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:193 [ 4128.873315] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4131.447599] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4133.859084] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4136.091728] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4137.951725] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4137.954528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.958590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4143.454932] Lustre: server umount lustre-OST0000 complete [ 4151.263132] Lustre: 266570:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325909/real 1761325909] req@00000000f2a50349 x1846884019741696/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325915 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4151.321651] Lustre: server umount lustre-MDT0000 complete [ 4156.180924] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 4157.099853] Key type lgssc unregistered [ 4157.212338] LNet: 267173:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4157.216324] LNet: Removed LNI 192.168.201.132@tcp [ 4157.501053] Key type .llcrypt unregistered [ 4157.502466] Key type ._llcrypt unregistered [ 4166.614755] alg: No test for adler32 (adler32-zlib) [ 4167.366356] Key type ._llcrypt registered [ 4167.367847] Key type .llcrypt registered [ 4167.398964] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4167.822201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4167.942322] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4168.013839] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 4168.015584] LNet: Accept secure, port 988 [ 4169.599114] Key type lgssc registered [ 4169.922989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4172.354137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4173.514386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4175.068726] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4177.549373] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4185.055238] Lustre: 269404:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325943/real 1761325943] req@00000000d770c018 x1846884100474880/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325949 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4185.116374] Lustre: server umount lustre-MDT0000 complete [ 4190.052962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4190.167327] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4190.180621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4190.253264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4191.587933] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4193.976369] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4196.456742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4196.523493] LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4196.528621] LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4196.532068] LustreError: 271200:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 4196.535508] LustreError: 271200:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-OST0000 [ 4196.537322] LustreError: 271200:0:(obd_mount_server.c:131:server_deregister_mount()) lustre-OST0000 not registered [ 4196.583482] Lustre: server umount lustre-OST0000 complete [ 4196.585156] LustreError: 271200:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4198.841507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4198.918967] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4198.983289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4200.529780] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4203.107964] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4204.002935] Lustre: lustre-OST0000: deleting orphan objects from 0x0:166 to 0x0:225 [ 4205.802050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4205.825915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4205.828348] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4205.854767] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4205.877582] Lustre: lustre-OST0001: new disk, initializing [ 4205.880396] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4205.902597] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4207.299193] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4209.787895] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4213.216682] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4213.216903] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.223941] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4216.817536] Lustre: server umount lustre-OST0001 complete [ 4218.336329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.336794] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4218.340573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4223.456063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4223.456580] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.219225] Lustre: server umount lustre-OST0000 complete [ 4230.623146] Lustre: 273749:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325989/real 1761325989] req@000000006859694b x1846884100485888/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325995 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4230.694929] Lustre: server umount lustre-MDT0000 complete [ 4234.450474] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 4235.417918] Key type lgssc unregistered [ 4235.525327] LNet: 274351:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4235.528631] LNet: Removed LNI 192.168.201.132@tcp [ 4235.845117] Key type .llcrypt unregistered [ 4235.846957] Key type ._llcrypt unregistered [ 4244.696415] alg: No test for adler32 (adler32-zlib) [ 4245.447444] Key type ._llcrypt registered [ 4245.448346] Key type .llcrypt registered [ 4245.496650] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 4251.149134] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4251.722397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4251.855654] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4251.931534] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 4251.933613] LNet: Accept secure, port 988 [ 4253.519126] Key type lgssc registered [ 4253.886521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4256.903429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4258.835474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4260.876560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4263.195023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4263.215983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.314389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4264.324659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4264.362486] Lustre: lustre-MDT0000: new disk, initializing [ 4264.390372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4264.398539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4265.764986] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4269.402956] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4271.522527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4271.542949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4271.545280] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4271.625176] Lustre: lustre-OST0000: new disk, initializing [ 4271.627381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4271.630869] Lustre: Skipped 1 previous similar message [ 4271.651858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.973482] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4276.505739] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4277.876229] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4277.947146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4278.752111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4278.752138] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4278.758301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4283.872583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4284.643066] Lustre: server umount lustre-OST0000 complete [ 4292.575163] Lustre: 280148:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326050/real 1761326050] req@00000000b38ad58a x1846884188559040/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326056 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4292.639943] Lustre: server umount lustre-MDT0000 complete [ 4295.511396] Lustre: DEBUG MARKER: == conf-sanity test 60: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 13:14:19 (1761326059) [ 4297.555482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4303.708526] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 4306.323195] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4309.389234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4311.539739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4313.815337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4316.290383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4316.313865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.406213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4316.418707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4316.457722] Lustre: lustre-MDT0000: new disk, initializing [ 4316.484988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.494787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4318.029916] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4321.996645] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4324.383253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4324.409394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4324.412114] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4324.504356] Lustre: lustre-OST0000: new disk, initializing [ 4324.507717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4324.511826] Lustre: Skipped 1 previous similar message [ 4324.536386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.139435] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4330.055524] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4331.420270] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4331.491597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4334.165092] Lustre: server umount lustre-OST0000 complete [ 4340.703154] Lustre: 285576:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326099/real 1761326099] req@00000000e531adf5 x1846884188564608/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761326105 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4346.847167] Lustre: 285576:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326105/real 1761326105] req@00000000ad21a8a6 x1846884188564928/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326111 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4346.903964] Lustre: server umount lustre-MDT0000 complete [ 4349.705300] Lustre: DEBUG MARKER: == conf-sanity test 61: large xattr ====================== 13:15:13 (1761326113) [ 4352.473655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.626090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.031438] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4356.662127] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4358.950700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4360.402893] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4362.883389] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4372.710462] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61.conf-sanity [ 4373.295478] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61.conf-sanity [ 4373.876937] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61.conf-sanity [ 4374.460196] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61.conf-sanity after remounting MDS [ 4375.163837] Lustre: Failing over lustre-MDT0000 [ 4375.348790] Lustre: server umount lustre-MDT0000 complete [ 4385.631218] Lustre: 276312:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326143/real 1761326143] req@000000004f65b0b1 x1846884188569664/t0(0) o400->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326150 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4385.639826] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4385.695274] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.661697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4390.879185] Lustre: 276311:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326148/real 1761326148] req@000000002ff8a248 x1846884188569856/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761326155 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4390.890432] Lustre: 276311:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4391.904595] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0x25f840717b25767a to 0x25f840717b25791a [ 4391.908813] Lustre: MGC192.168.201.132@tcp: Connection restored to 192.168.201.132@tcp (at 0@lo) [ 4392.040507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.043235] Lustre: Skipped 1 previous similar message [ 4392.065856] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4393.055588] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4393.066818] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4393.085460] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 4393.582893] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4397.476211] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4397.976290] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4399.840280] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61.conf-sanity [ 4400.543532] Lustre: Failing over lustre-MDT0000 [ 4400.847067] Lustre: server umount lustre-MDT0000 complete [ 4408.287158] Lustre: 276311:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326165/real 1761326165] req@00000000c4e0314e x1846884188573568/t0(0) o400->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326172 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4408.293491] Lustre: 276311:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4408.295685] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4414.186685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.432893] Lustre: Evicted from MGS (at 192.168.201.132@tcp) after server handle changed from 0x25f840717b25791a to 0x25f840717b257b90 [ 4414.439815] Lustre: MGC192.168.201.132@tcp: Connection restored to (at 0@lo) [ 4414.564783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4414.590656] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4416.042347] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4416.096423] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 4416.106060] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 4416.121297] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:97 [ 4418.658899] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4419.676029] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4420.252047] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4424.160045] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4424.165098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4428.668775] Lustre: server umount lustre-MDT0000 complete [ 4430.007994] LustreError: 288034:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326194 with bad export cookie 2736007629769112464 [ 4430.009708] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4430.011368] LustreError: 288034:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4434.751955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.439058] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4438.966176] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4441.306388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4442.979423] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4445.551682] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4449.571127] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:129 [ 4454.880186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.880798] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4454.884661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4464.607150] Lustre: 293590:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326223/real 1761326223] req@0000000063573cab x1846884188582720/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326229 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4464.669612] Lustre: server umount lustre-MDT0000 complete [ 4464.671197] Lustre: Skipped 2 previous similar messages [ 4469.686377] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 4470.739772] Key type lgssc unregistered [ 4470.863457] LNet: 294193:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4470.867481] LNet: Removed LNI 192.168.201.132@tcp [ 4471.174881] Key type .llcrypt unregistered [ 4471.176552] Key type ._llcrypt unregistered [ 4480.348565] alg: No test for adler32 (adler32-zlib) [ 4481.100317] Key type ._llcrypt registered [ 4481.101219] Key type .llcrypt registered [ 4481.135950] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4481.467178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4481.655352] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4481.719467] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 4481.722070] LNet: Accept secure, port 988 [ 4483.303162] Key type lgssc registered [ 4483.632770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4486.351361] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4486.353903] LustreError: 295675:0:(osd_handler.c:8259:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 4486.362995] LustreError: 295675:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 4486.365599] LustreError: 295675:0:(obd_mount.c:200:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 4486.367609] LustreError: 295675:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 4486.370703] LustreError: 295675:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4488.991145] LDISKFS-fs (dm-1): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.993668] LustreError: 296071:0:(osd_handler.c:8259:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 4489.002189] LustreError: 296071:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-22) [ 4489.004613] LustreError: 296071:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 4489.006734] LustreError: 296071:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 4489.009101] LustreError: 296071:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4494.174555] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 4495.286621] Key type lgssc unregistered [ 4495.408425] LNet: 296875:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4495.412184] LNet: Removed LNI 192.168.201.132@tcp [ 4495.732798] Key type .llcrypt unregistered [ 4495.734436] Key type ._llcrypt unregistered [ 4501.418905] alg: No test for adler32 (adler32-zlib) [ 4502.171287] Key type ._llcrypt registered [ 4502.172370] Key type .llcrypt registered [ 4502.208619] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 4508.246882] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4508.610851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4508.860802] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4508.943702] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 4508.945842] LNet: Accept secure, port 988 [ 4510.527106] Key type lgssc registered [ 4510.938522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4514.233648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4516.365908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4518.502190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4521.030593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4521.060694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4522.202666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4522.215990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4522.261892] Lustre: lustre-MDT0000: new disk, initializing [ 4522.288422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.295972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4523.967659] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4528.060324] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4530.496377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4530.525521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4530.530249] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4530.649603] Lustre: lustre-OST0000: new disk, initializing [ 4530.653241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4530.656013] Lustre: Skipped 1 previous similar message [ 4530.657236] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4530.682742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4532.220512] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4536.213456] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4537.928726] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4538.028776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4540.897052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.902078] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4540.902520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4544.881158] Lustre: server umount lustre-OST0000 complete [ 4551.647148] Lustre: 302075:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326310/real 1761326310] req@000000003b6be685 x1846884458043584/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326316 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4551.735746] Lustre: server umount lustre-MDT0000 complete [ 4554.613876] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 4555.221660] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:18:39 (1761326319) [ 4557.655392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.859339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4559.463126] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4562.180253] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4564.543381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4564.657836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4566.025494] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4568.376470] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4570.693757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4570.721044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4570.724229] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4570.773602] Lustre: lustre-OST0001: new disk, initializing [ 4570.775459] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4570.797202] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4572.256321] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4576.002315] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4579.892342] Lustre: server umount lustre-OST0001 complete [ 4580.591642] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.201.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4581.856974] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.857055] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4581.861389] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.976780] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4592.096572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4592.096571] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4592.097046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4597.215914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4597.216248] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4598.119507] Lustre: server umount lustre-OST0000 complete [ 4612.063210] Lustre: 305914:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326370/real 1761326370] req@00000000590303f7 x1846884458054080/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326376 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4612.127261] Lustre: server umount lustre-MDT0000 complete [ 4616.875743] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 4617.903530] Key type lgssc unregistered [ 4618.031472] LNet: 306517:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4618.034919] LNet: Removed LNI 192.168.201.132@tcp [ 4618.344032] Key type .llcrypt unregistered [ 4618.345730] Key type ._llcrypt unregistered [ 4627.184557] alg: No test for adler32 (adler32-zlib) [ 4627.936399] Key type ._llcrypt registered [ 4627.937791] Key type .llcrypt registered [ 4627.976860] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 4633.695568] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4634.148983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4634.259664] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4634.338504] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 4634.339980] LNet: Accept secure, port 988 [ 4635.919108] Key type lgssc registered [ 4636.234104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4638.974980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4640.858271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4642.818250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4645.005467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4645.028322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.119446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4646.130431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4646.171420] Lustre: lustre-MDT0000: new disk, initializing [ 4646.193224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4646.198988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4647.540324] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4651.299644] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4653.470295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4653.488743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4653.491367] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4653.575843] Lustre: lustre-OST0000: new disk, initializing [ 4653.578701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4653.581551] Lustre: Skipped 1 previous similar message [ 4653.600884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.986716] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4658.834489] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4660.270432] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4660.336086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4661.216313] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4661.216670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.222995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4666.335962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4667.042428] Lustre: server umount lustre-OST0000 complete [ 4674.527135] Lustre: 312315:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326432/real 1761326432] req@00000000de778165 x1846884589115136/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326438 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4674.594140] Lustre: server umount lustre-MDT0000 complete [ 4677.336333] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 13:20:41 (1761326441) [ 4679.334638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4681.961052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.092954] Lustre: lustre-MDT0000: new disk, initializing [ 4682.118476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4683.483230] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4685.966873] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4692.959243] Lustre: 314143:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326451/real 1761326451] req@0000000038a385f3 x1846884589117184/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326457 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4693.022275] Lustre: server umount lustre-MDT0000 complete [ 4696.114913] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:21:00 (1761326460) [ 4698.333278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.493975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.906296] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4702.339830] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4704.584852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4704.699539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.042356] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4708.531662] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4717.626855] Lustre: Permanently deactivating lustre-OST0000 [ 4717.631558] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [ 4726.767383] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4727.203145] LustreError: 317082:0:(mgs_llog.c:1551:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [ 4727.206468] LustreError: 317082:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 4727.209174] LustreError: 317082:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 4727.825914] LustreError: 317130:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 4727.828658] LustreError: 317130:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 4734.556880] Lustre: server umount lustre-OST0000 complete [ 4735.440848] LustreError: 317380:0:(mgs_llog.c:1605:mgs_replace_nids()) Only MGS is allowed to be started [ 4735.443234] LustreError: 317380:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -115 [ 4742.111107] Lustre: 317478:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326500/real 1761326500] req@00000000526d3f77 x1846884589124672/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761326506 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4748.255218] Lustre: 317478:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326506/real 1761326506] req@00000000afc01d07 x1846884589124928/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326512 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4748.334574] Lustre: server umount lustre-MDT0000 complete [ 4751.092108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4752.558749] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4753.473135] LustreError: 318387:0:(mgs_handler.c:1069:mgs_iocontrol()) MGS: error replacing nids: rc = -6 [ 4755.524813] Lustre: 318731:0:(mgs_llog.c:1311:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.132@tcp [ 4761.567123] Lustre: 318829:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326520/real 1761326520] req@00000000ad2d56a0 x1846884589125056/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326526 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4761.577906] Lustre: server umount MGS complete [ 4764.413027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.563121] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4764.581632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.063749] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4768.529561] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4769.167963] Lustre: Permanently reactivating lustre-OST0000 [ 4769.171689] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 4769.173652] Lustre: Skipped 1 previous similar message [ 4775.447317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.916377] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4779.246961] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4788.192849] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4788.193169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4788.199690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4792.032458] Lustre: server umount lustre-OST0000 complete [ 4799.455197] Lustre: 321483:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326557/real 1761326557] req@0000000028455296 x1846884589131584/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326563 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4804.508878] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 4805.642316] Key type lgssc unregistered [ 4805.759479] LNet: 322085:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4805.762258] LNet: Removed LNI 192.168.201.132@tcp [ 4806.041882] Key type .llcrypt unregistered [ 4806.043168] Key type ._llcrypt unregistered [ 4811.352110] alg: No test for adler32 (adler32-zlib) [ 4812.103519] Key type ._llcrypt registered [ 4812.105164] Key type .llcrypt registered [ 4812.139196] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 4818.292499] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4818.832925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4818.995872] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4819.079633] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 4819.081499] LNet: Accept secure, port 988 [ 4820.671114] Key type lgssc registered [ 4821.041907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4824.210487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4826.307710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4828.556324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4830.912313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4830.933591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.040592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4832.054444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4832.102885] Lustre: lustre-MDT0000: new disk, initializing [ 4832.129528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4832.135761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4833.701539] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4837.584459] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4839.966622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4839.991208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4839.996127] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4840.093151] Lustre: lustre-OST0000: new disk, initializing [ 4840.094955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4840.096629] Lustre: Skipped 1 previous similar message [ 4840.119956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.636370] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4845.554371] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4847.216472] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4847.309323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4851.680684] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4851.682864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.686572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4854.046602] Lustre: server umount lustre-OST0000 complete [ 4861.407146] Lustre: 327272:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326619/real 1761326619] req@00000000671dee44 x1846884783101952/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326625 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4861.478317] Lustre: server umount lustre-MDT0000 complete [ 4864.299087] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 4864.939695] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:23:49 (1761326629) [ 4867.313200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.490884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4868.862265] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4871.451170] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4873.835353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4873.941651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4875.466821] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4878.143302] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4878.843258] Lustre: ctl-lustre-MDT0000: [0x0000000240000400-0x0000000280000400]:0:mdt sequences allocated: rc = 0 [ 4885.658109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:mdt [ 4888.544296] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4888.544422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4888.551432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4893.032161] Lustre: server umount lustre-OST0000 complete [ 4900.319123] Lustre: 330070:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326658/real 1761326658] req@00000000b99e9951 x1846884783107072/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326664 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4900.378109] Lustre: server umount lustre-MDT0000 complete [ 4905.075804] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 4906.139990] Key type lgssc unregistered [ 4906.270519] LNet: 330672:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4906.273601] LNet: Removed LNI 192.168.201.132@tcp [ 4906.558587] Key type .llcrypt unregistered [ 4906.559622] Key type ._llcrypt unregistered [ 4924.720422] alg: No test for adler32 (adler32-zlib) [ 4925.472300] Key type ._llcrypt registered [ 4925.473287] Key type .llcrypt registered [ 4925.507875] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 4925.598314] Key type .llcrypt unregistered [ 4925.600020] Key type ._llcrypt unregistered [ 4931.529742] alg: No test for adler32 (adler32-zlib) [ 4932.281305] Key type ._llcrypt registered [ 4932.282269] Key type .llcrypt registered [ 4932.321246] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 4932.644693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4932.867602] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4932.938266] LNet: Added LNI 192.168.201.132@tcp [8/256/0/180] [ 4932.940487] LNet: Accept secure, port 988 [ 4934.519135] Key type lgssc registered [ 4934.924355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4937.548390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4939.473495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4941.689974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4941.714721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4942.801173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4942.811797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4942.851572] Lustre: lustre-MDT0000: new disk, initializing [ 4942.875581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4942.883836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4944.320300] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4948.079884] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4950.383749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4950.410113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4950.412843] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4950.519420] Lustre: lustre-OST0000: new disk, initializing [ 4950.523378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4950.526693] Lustre: Skipped 1 previous similar message [ 4950.550902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.956315] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4955.832440] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4962.272755] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.276324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4964.302553] Lustre: server umount lustre-MDT0000 complete [ 4965.674288] LustreError: 336003:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761326730 with bad export cookie 18135972425984439309 [ 4965.676194] LustreError: 166-1: MGC192.168.201.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4965.677757] LustreError: 336003:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4965.743658] Lustre: server umount lustre-OST0000 complete [ 4969.250356] Lustre: DEBUG MARKER: == conf-sanity test 73: failnode to update from mountdata properly ========================================================== 13:25:33 (1761326733) [ 4969.604427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4971.797536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4971.960185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4973.252208] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4975.653129] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4977.840273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4977.866017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4977.938064] Lustre: Found index 0 for lustre-OST0000, updating log [ 4977.979210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4979.392210] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 4981.875896] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4987.107034] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5 to 0x0:33 [ 4992.481444] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4992.481455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4992.487600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4994.665563] Lustre: server umount lustre-OST0000 complete [ 5003.743133] Lustre: 339010:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326762/real 1761326762] req@00000000c1c99bea x1846884902645440/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326768 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5003.806065] Lustre: server umount lustre-MDT0000 complete [ 5006.579053] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 13:26:10 (1761326770) [ 5008.498983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5010.147288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5011.915181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5013.462280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5017.862455] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 5020.821198] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 5023.978764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5025.962861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5027.903778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5030.011585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5030.034185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5030.107130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5030.117380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5030.153541] Lustre: lustre-MDT0000: new disk, initializing [ 5030.175468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5030.181758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5031.541037] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 5035.320016] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5037.618609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5037.648757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5037.651166] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5037.735756] Lustre: lustre-OST0000: new disk, initializing [ 5037.737583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5037.740131] Lustre: Skipped 1 previous similar message [ 5039.089749] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_default_debug -1 all 8 [ 5042.709885] Lustre: DEBUG MARKER: oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5044.113974] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5044.188987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5046.883628] Lustre: server umount lustre-OST0000 complete [ 5054.431136] Lustre: 344874:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326812/real 1761326812] req@00000000f0e826d7 x1846884902651072/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761326818 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5060.575148] Lustre: 344874:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761326818/real 1761326818] req@000000001b74b819 x1846884902651392/t0(0) o251->MGC192.168.201.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761326824 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5060.644192] Lustre: server umount lustre-MDT0000 complete [ 5067.630817] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing set_hostid [ 5070.620868] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing load_modules_local [ 5074.299413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5076.484406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5078.669782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5079.212399] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4942 sec ========== 13:27:23 (1761326843) [ 5113.731852] Lustre: DEBUG MARKER: oleg132-server.virtnet: executing unload_modules_local [ 5114.789819] Key type lgssc unregistered [ 5114.908448] LNet: 348925:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5114.912524] LNet: Removed LNI 192.168.201.132@tcp [ 5115.237916] Key type .llcrypt unregistered [ 5115.238993] Key type ._llcrypt unregistered