[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478289632 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] BRK [0x5ee01000, 0x5ee01fff] PGTABLE [ 0.000000] BRK [0x5ee02000, 0x5ee02fff] PGTABLE [ 0.000000] BRK [0x5ee03000, 0x5ee03fff] PGTABLE [ 0.000000] BRK [0x5ee04000, 0x5ee04fff] PGTABLE [ 0.000000] BRK [0x5ee05000, 0x5ee05fff] PGTABLE [ 0.000000] BRK [0x5ee06000, 0x5ee06fff] PGTABLE [ 0.000000] BRK [0x5ee07000, 0x5ee07fff] PGTABLE [ 0.000000] BRK [0x5ee08000, 0x5ee08fff] PGTABLE [ 0.000000] BRK [0x5ee09000, 0x5ee09fff] PGTABLE [ 0.000000] BRK [0x5ee0a000, 0x5ee0afff] PGTABLE [ 0.000000] BRK [0x5ee0b000, 0x5ee0bfff] PGTABLE [ 0.000000] BRK [0x5ee0c000, 0x5ee0cfff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] On node 0 totalpages: 1076588 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 158 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:63 [ 0.000000] Normal zone: 4536 pages used for memmap [ 0.000000] Normal zone: 290304 pages, LIFO batch:63 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] pcpu-alloc: s221184 r8192 d28672 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001000] hpet clockevent registered [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002401] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.006924] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009149] LSM: Security Framework initializing [ 0.010060] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.012071] *** VALIDATE selinux *** [ 0.020405] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024732] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028106] *** VALIDATE tmpfs *** [ 0.029398] *** VALIDATE proc *** [ 0.031046] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033231] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036034] Spectre V2 : User space: Vulnerable [ 0.037012] Speculative Store Bypass: Vulnerable [ 0.040424] debug: unmapping init [mem 0xffffffff87459000-0xffffffff87460fff] [ 0.042144] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043653] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.045021] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051296] rcu: Hierarchical SRCU implementation. [ 0.053431] smp: Bringing up secondary CPUs ... [ 0.054590] x86: Booting SMP configuration: [ 0.055026] .... node #0, CPUs: #1 #2 #3 [ 0.058223] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144615] node 0 deferred pages initialised in 80ms [ 0.147327] devtmpfs: initialized [ 0.149264] x86/mm: Memory block size: 128MB [ 0.151877] gcov: version magic: 0x41383552 [ 0.155207] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156155] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157000] pinctrl core: initialized pinctrl subsystem [ 0.158021] ************************************************************* [ 0.159000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162017] ** ** [ 0.164019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166019] ** ** [ 0.168020] ** This means that this kernel is built to expose internal ** [ 0.171019] ** IOMMU data structures, which may compromise security on ** [ 0.172016] ** your system. ** [ 0.175020] ** ** [ 0.177013] ** If you see this message and you are not debugging the ** [ 0.179020] ** kernel, report this immediately to your vendor! ** [ 0.182021] ** ** [ 0.184018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187037] ************************************************************* [ 0.189847] NET: Registered protocol family 16 [ 0.191505] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196113] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199101] cpuidle: using governor menu [ 0.201499] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203546] PCI: Using configuration type 1 for base access [ 0.206113] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214114] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220059] cryptd: max_cpu_qlen set to 1000 [ 0.225042] ACPI: Added _OSI(Module Device) [ 0.227013] ACPI: Added _OSI(Processor Device) [ 0.228014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.234558] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241402] ACPI: Interpreter enabled [ 0.242068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244015] ACPI: Using IOAPIC for interrupt routing [ 0.245094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249402] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258229] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267283] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.271210] acpiphp: Slot [2] registered [ 0.273178] acpiphp: Slot [3] registered [ 0.274092] acpiphp: Slot [4] registered [ 0.276121] acpiphp: Slot [5] registered [ 0.277163] acpiphp: Slot [6] registered [ 0.279099] acpiphp: Slot [7] registered [ 0.280112] acpiphp: Slot [8] registered [ 0.282114] acpiphp: Slot [9] registered [ 0.283102] acpiphp: Slot [10] registered [ 0.285134] acpiphp: Slot [11] registered [ 0.286107] acpiphp: Slot [12] registered [ 0.288102] acpiphp: Slot [13] registered [ 0.289104] acpiphp: Slot [14] registered [ 0.291095] acpiphp: Slot [15] registered [ 0.292114] acpiphp: Slot [16] registered [ 0.294106] acpiphp: Slot [17] registered [ 0.295105] acpiphp: Slot [18] registered [ 0.297101] acpiphp: Slot [19] registered [ 0.298100] acpiphp: Slot [20] registered [ 0.300100] acpiphp: Slot [21] registered [ 0.301100] acpiphp: Slot [22] registered [ 0.303100] acpiphp: Slot [23] registered [ 0.304140] acpiphp: Slot [24] registered [ 0.306116] acpiphp: Slot [25] registered [ 0.307098] acpiphp: Slot [26] registered [ 0.309105] acpiphp: Slot [27] registered [ 0.310108] acpiphp: Slot [28] registered [ 0.312071] acpiphp: Slot [29] registered [ 0.313117] acpiphp: Slot [30] registered [ 0.315129] acpiphp: Slot [31] registered [ 0.317061] PCI host bridge to bus 0000:00 [ 0.318018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.320017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.322020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.324021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.326019] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.329028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.330116] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334301] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339586] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.349014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.362019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364286] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367671] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372611] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.393028] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.399025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405457] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.418015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.436016] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.448102] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.460018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.479018] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.487219] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.493019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.497019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.508020] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.516788] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.526022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.534027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.547023] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.558661] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.564019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.569019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.581018] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.592249] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.603022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.609025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.623024] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.636409] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.640616] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.642543] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.645483] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.647268] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.652141] iommu: Default domain type: Passthrough [ 0.653000] SCSI subsystem initialized [ 0.653189] ACPI: bus type USB registered [ 0.655128] usbcore: registered new interface driver usbfs [ 0.657125] usbcore: registered new interface driver hub [ 0.658134] usbcore: registered new device driver usb [ 0.660220] pps_core: LinuxPPS API ver. 1 registered [ 0.662014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.666080] PTP clock support registered [ 0.668059] EDAC MC: Ver: 3.0.0 [ 0.669533] PCI: Using ACPI for IRQ routing [ 0.671021] PCI: pci_cache_line_size set to 64 bytes [ 0.671339] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.671357] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.671360] e820: reserve RAM buffer [mem 0x146e00000-0x147ffffff] [ 0.671964] NetLabel: Initializing [ 0.673017] NetLabel: domain hash size = 128 [ 0.675016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.677105] NetLabel: unlabeled traffic allowed by default [ 0.679164] vgaarb: loaded [ 0.682065] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.684019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.689230] clocksource: Switched to clocksource kvm-clock [ 0.800326] VFS: Disk quotas dquot_6.6.0 [ 0.801882] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.804267] *** VALIDATE ramfs *** [ 0.805425] *** VALIDATE hugetlbfs *** [ 0.806891] pnp: PnP ACPI init [ 0.808155] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.808940] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.808970] pnp 00:02: [dma 2] [ 0.809051] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.809126] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.809189] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.809264] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.809866] pnp: PnP ACPI: found 6 devices [ 0.834978] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.838052] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.840116] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.842644] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.844849] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.847365] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.849991] NET: Registered protocol family 2 [ 0.852757] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.857199] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.860883] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.866157] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.869762] TCP: Hash tables configured (established 65536 bind 65536) [ 0.872767] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.876340] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.879532] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.882672] NET: Registered protocol family 1 [ 0.886248] RPC: Registered named UNIX socket transport module. [ 0.888393] RPC: Registered udp transport module. [ 0.890215] RPC: Registered tcp transport module. [ 0.892042] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.894341] NET: Registered protocol family 44 [ 0.895861] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.897839] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.899671] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.902022] PCI: CLS 0 bytes, default 64 [ 0.903714] Unpacking initramfs... [ 2.295615] debug: unmapping init [mem 0xffff89823cc54000-0xffff89823ffbffff] [ 2.299503] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.302058] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.305390] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.827104] Initialise system trusted keyrings [ 2.829072] Key type blacklist registered [ 2.832088] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.841548] zbud: loaded [ 2.844496] *** VALIDATE nfs *** [ 2.846012] *** VALIDATE nfs4 *** [ 2.847577] pstore: using deflate compression [ 2.850655] Platform Keyring initialized [ 2.952300] NET: Registered protocol family 38 [ 2.954288] Key type asymmetric registered [ 2.955845] Asymmetric key parser 'x509' registered [ 2.957984] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.962179] io scheduler mq-deadline registered [ 2.963667] io scheduler kyber registered [ 2.965257] io scheduler bfq registered [ 2.967403] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.970615] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.973722] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.977050] ACPI: Power Button [PWRF] [ 3.067313] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.160483] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.356909] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.451110] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.647679] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.680891] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.717513] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.723182] Non-volatile memory driver v1.3 [ 3.725075] Linux agpgart interface v0.103 [ 3.759042] virtio_blk virtio1: [vda] 67984 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.763042] vda: detected capacity change from 0 to 34807808 [ 3.778268] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.781631] vdb: detected capacity change from 0 to 1073741824 [ 3.797203] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.800247] vdc: detected capacity change from 0 to 2621440000 [ 3.816388] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.819368] vdd: detected capacity change from 0 to 2621440000 [ 3.832745] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.834890] vde: detected capacity change from 0 to 4294967296 [ 3.848104] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.851119] vdf: detected capacity change from 0 to 4294967296 [ 3.860429] libphy: Fixed MDIO Bus: probed [ 3.868514] usbcore: registered new interface driver usbserial_generic [ 3.871319] usbserial: USB Serial support registered for generic [ 3.873552] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.877151] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.878654] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.880707] mousedev: PS/2 mouse device common for all mice [ 3.883593] rtc_cmos 00:05: RTC can wake from S4 [ 3.886753] rtc_cmos 00:05: registered as rtc0 [ 3.887063] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.888718] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.895195] intel_pstate: CPU model not supported [ 3.898416] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.898720] hid: raw HID events driver (C) Jiri Kosina [ 3.903104] usbcore: registered new interface driver usbhid [ 3.904714] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.905322] usbhid: USB HID core driver [ 3.905467] drop_monitor: Initializing network drop monitor service [ 3.911413] Initializing XFRM netlink socket [ 3.913353] NET: Registered protocol family 10 [ 3.916121] Segment Routing with IPv6 [ 3.917307] NET: Registered protocol family 17 [ 3.919179] mpls_gso: MPLS GSO support [ 3.920751] start plist test [ 3.922993] end plist test [ 3.924499] RAS: Correctable Errors collector initialized. [ 3.926212] AVX version of gcm_enc/dec engaged. [ 3.927634] AES CTR mode by8 optimization enabled [ 4.001124] sched_clock: Marking stable (4001101341, 0)->(4915675084, -914573743) [ 4.005322] registered taskstats version 1 [ 4.007294] Loading compiled-in X.509 certificates [ 4.009530] zswap: loaded using pool lzo/zbud [ 4.033212] Key type big_key registered [ 4.046360] Key type encrypted registered [ 4.048214] ima: No TPM chip found, activating TPM-bypass! [ 4.050313] ima: Allocated hash algorithm: sha1 [ 4.051540] ima: No architecture policies found [ 4.053085] evm: Initialising EVM extended attributes: [ 4.054810] evm: security.selinux [ 4.056025] evm: security.ima [ 4.057162] evm: security.capability [ 4.058170] evm: HMAC attrs: 0x1 [ 4.060487] rtc_cmos 00:05: setting system clock to 2025-10-24 13:46:26 UTC (1761313586) [ 4.066748] debug: unmapping init [mem 0xffffffff88403000-0xffffffff885fffff] [ 4.069652] debug: unmapping init [mem 0xffffffff87182000-0xffffffff87458fff] [ 4.079026] Write protecting the kernel read-only data: 28672k [ 4.082043] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 4.084936] debug: unmapping init [mem 0xffffffff86114000-0xffffffff861fffff] [ 4.121305] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.130216] systemd[1]: Detected virtualization kvm. [ 4.132085] systemd[1]: Detected architecture x86-64. [ 4.134083] systemd[1]: Running in initial RAM disk. [ 4.166899] systemd[1]: No hostname configured. [ 4.168776] systemd[1]: Set hostname to . [ 4.171063] random: systemd: uninitialized urandom read (16 bytes read) [ 4.173682] systemd[1]: Initializing machine ID from random generator. [ 4.316047] random: systemd: uninitialized urandom read (16 bytes read) [ 4.318661] systemd[1]: Reached target Local File Systems. [ 4.324603] random: systemd: uninitialized urandom read (16 bytes read) [ 4.327835] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ 4.334460] systemd[1]: Reached target Local Encrypted Volumes. [ 5.014900] device-mapper: uevent: version 1.0.3 [ 5.017685] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 5.816668] random: fast init done [ 5.852227] virtio_net virtio0 ens2: renamed from eth0 [ 5.858047] libata version 3.00 loaded. [ 5.860986] ata_piix 0000:00:01.1: version 2.13 [ 5.939899] scsi host0: ata_piix [ 5.955663] scsi host1: ata_piix [ 5.957801] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.960435] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.563786] random: crng init done [ 10.564935] random: 7 urandom warning(s) missed due to ratelimiting [ 11.110276] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 12.332408] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.595894] SELinux: Disabled at runtime. [ 12.658427] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.665355] systemd[1]: Detected virtualization kvm. [ 12.667215] systemd[1]: Detected architecture x86-64. [ 13.206966] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.210622] systemd[1]: Stopped Switch Root. [ 13.217029] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.221278] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.224653] systemd[1]: Stopped Journal Service. [ 13.241581] systemd[1]: Starting Journal Service... [ 13.255465] systemd[1]: Mounting Huge Pages File System... [ 13.407348] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 13.711615] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 14.061802] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.080813] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.286659] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.296744] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 14.296770] EDAC sbridge: Ver: 1.1.2 [ 15.868833] Key type dns_resolver registered [ 16.177485] NFS: Registering the id_resolver key type [ 16.179173] Key type id_resolver registered [ 16.180485] Key type id_legacy registered [ 38.539273] libcfs: loading out-of-tree module taints kernel. [ 38.561151] alg: No test for adler32 (adler32-zlib) [ 39.315463] Key type ._llcrypt registered [ 39.316946] Key type .llcrypt registered [ 39.367355] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 48.760577] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 49.916330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.429058] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 50.970663] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 50.973966] LNet: Accept secure, port 988 [ 52.656128] Key type lgssc registered [ 53.794578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 63.658558] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 65.141052] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 71.583060] hrtimer: interrupt took 6052675 ns [ 71.904439] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.297508] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.579805] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.273540] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 99.734428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.776497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 100.951962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 100.983734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 101.048166] Lustre: lustre-MDT0000: new disk, initializing [ 101.132546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 101.147523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 103.839285] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 112.032743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.091281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 112.136496] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 112.152741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 112.156451] Lustre: Skipped 1 previous similar message [ 112.211489] Lustre: lustre-MDT0001: new disk, initializing [ 112.266543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 112.286955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 112.298564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 114.934525] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 123.997614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.050385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 124.054643] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 124.230942] Lustre: lustre-OST0000: new disk, initializing [ 124.235817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 124.290783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.949068] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 132.116584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 132.131313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 136.373324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.433491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 136.440126] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 136.514446] Lustre: lustre-OST0001: new disk, initializing [ 136.521109] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 136.563971] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 139.418477] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 141.316098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 141.327626] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 147.486768] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 153.875293] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 161.236764] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing check_logdir /tmp/testlogs/ [ 163.971342] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing yml_node [ 166.727376] Lustre: DEBUG MARKER: Client: 2.15.7.10 [ 168.337156] Lustre: DEBUG MARKER: MDS: 2.15.7.10 [ 169.741983] Lustre: DEBUG MARKER: OSS: 2.15.7.10 [ 170.629595] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 09:49:12 EDT 2025 [ 174.922331] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball 110 [ 175.704610] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 189.410307] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 189.413571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 189.421032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 192.482733] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 192.487265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 192.490784] Lustre: Skipped 2 previous similar messages [ 192.506433] Lustre: Skipped 3 previous similar messages [ 197.601520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 202.727956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 202.734043] Lustre: Skipped 6 previous similar messages [ 203.232765] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 203.318686] Lustre: server umount lustre-MDT0000 complete [ 205.777217] LustreError: 7237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313788 with bad export cookie 17008893997869763137 [ 205.777585] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 205.781524] LustreError: 7237:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 207.841417] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 207.845633] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 207.845908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 207.856051] LustreError: Skipped 1 previous similar message [ 207.863831] Lustre: Skipped 1 previous similar message [ 210.849051] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 210.856760] LustreError: Skipped 1 previous similar message [ 211.875417] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 211.881701] LustreError: Skipped 1 previous similar message [ 217.059483] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 217.059596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 217.067983] LustreError: Skipped 1 previous similar message [ 217.072792] Lustre: Skipped 6 previous similar messages [ 220.128159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 220.413625] Lustre: server umount lustre-MDT0001 complete [ 223.563922] Lustre: server umount lustre-OST0000 complete [ 226.843300] Lustre: server umount lustre-OST0001 complete [ 231.541312] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 237.406343] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 244.860498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 249.409273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 253.078337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 257.007690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 261.168760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 261.217284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 261.357644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 261.381412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 261.446275] Lustre: lustre-MDT0000: new disk, initializing [ 261.527782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 261.541603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 264.046369] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 271.916482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 271.963753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 272.023404] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 272.050166] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 272.054159] Lustre: Skipped 1 previous similar message [ 272.132135] Lustre: lustre-MDT0001: new disk, initializing [ 272.213236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.243355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 272.254577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 275.012295] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 283.297883] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 287.771952] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 292.399661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 292.458570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 292.462440] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 292.635552] Lustre: lustre-OST0000: new disk, initializing [ 292.639917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 292.714845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 294.498065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 294.505849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 295.730979] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 302.471462] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 305.400943] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 305.627903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 308.891433] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 309.044958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 315.361887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.363209] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 315.365644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 315.369038] Lustre: Skipped 1 previous similar message [ 316.641807] Lustre: server umount lustre-OST0000 complete [ 320.483232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.478588] Lustre: server umount lustre-MDT0000 complete [ 325.604956] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.612089] LustreError: Skipped 1 previous similar message [ 327.803402] LustreError: 14300:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313910 with bad export cookie 17008893997869764586 [ 327.808328] LustreError: 14300:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 327.811680] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 330.733694] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:51:52 (1761313912) [ 332.517514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 333.809401] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:51:55 (1761313915) [ 338.219201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.483966] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.522968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 341.403454] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 347.495469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.779159] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 356.281956] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 359.860253] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 363.932797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 367.290791] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 372.915821] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 382.951793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.956981] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 382.957205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 382.957210] Lustre: Skipped 3 previous similar messages [ 382.966578] Lustre: Skipped 2 previous similar messages [ 382.971090] LustreError: Skipped 1 previous similar message [ 383.886178] Lustre: server umount lustre-OST0000 complete [ 383.891934] Lustre: Skipped 1 previous similar message [ 401.376113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 404.319403] LustreError: 18835:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313986 with bad export cookie 17008893997869765447 [ 404.320223] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 404.325678] LustreError: 18835:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 414.559166] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 416.916375] Key type lgssc unregistered [ 417.125815] LNet: 22507:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.132790] LNet: Removed LNI 192.168.203.123@tcp [ 417.782234] Key type .llcrypt unregistered [ 417.784121] Key type ._llcrypt unregistered [ 435.216410] alg: No test for adler32 (adler32-zlib) [ 435.968388] Key type ._llcrypt registered [ 435.972569] Key type .llcrypt registered [ 436.056780] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 436.890560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 437.376418] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 437.548799] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 437.553700] LNet: Accept secure, port 988 [ 439.208113] Key type lgssc registered [ 440.114838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 446.517636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.806253] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 447.836548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.719824] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 453.090863] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.866270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.088352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 460.392479] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 465.794358] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 469.516964] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 473.953064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.219188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.218775] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 478.316581] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 482.193085] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 493.029033] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.029874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.036470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.043733] Lustre: Skipped 1 previous similar message [ 494.487132] Lustre: server umount lustre-OST0000 complete [ 498.146035] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 498.149540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.158671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 498.158679] Lustre: Skipped 1 previous similar message [ 498.180355] Lustre: Skipped 1 previous similar message [ 503.265567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.268497] Lustre: Skipped 1 previous similar message [ 503.417253] Lustre: server umount lustre-MDT0000 complete [ 505.952213] LustreError: 26613:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314088 with bad export cookie 13883676925589975226 [ 505.953578] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.957278] LustreError: 26613:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 506.170669] Lustre: server umount lustre-MDT0001 complete [ 514.538033] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 516.878807] Key type lgssc unregistered [ 517.119548] LNet: 27619:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.125225] LNet: Removed LNI 192.168.203.123@tcp [ 517.776394] Key type .llcrypt unregistered [ 517.777888] Key type ._llcrypt unregistered [ 533.229551] alg: No test for adler32 (adler32-zlib) [ 533.982528] Key type ._llcrypt registered [ 533.984629] Key type .llcrypt registered [ 534.053343] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 534.750141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 534.958749] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 535.070510] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 535.072955] LNet: Accept secure, port 988 [ 536.704153] Key type lgssc registered [ 537.480620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 543.010109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.336035] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.378766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.377788] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 549.862138] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.946830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.186651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 556.851828] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 561.848544] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 565.303191] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 569.481871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 569.689985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.012449] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 574.832884] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 577.576274] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 589.286442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.289429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 589.296853] Lustre: Skipped 1 previous similar message [ 589.310025] Lustre: Skipped 2 previous similar messages [ 592.411801] Lustre: server umount lustre-MDT0000 complete [ 594.402070] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.422821] LustreError: Skipped 2 previous similar messages [ 595.302513] LustreError: 29061:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314177 with bad export cookie 16246018163103439825 [ 595.305852] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.311437] LustreError: 29061:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 599.521781] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.524236] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.533415] LustreError: Skipped 1 previous similar message [ 599.548872] Lustre: Skipped 1 previous similar message [ 599.575438] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 599.578244] Lustre: Skipped 1 previous similar message [ 601.689239] Lustre: server umount lustre-MDT0001 complete [ 630.240165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 630.391845] Lustre: server umount lustre-OST0000 complete [ 643.783726] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 646.238996] Key type lgssc unregistered [ 646.552809] LNet: 33063:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.562693] LNet: Removed LNI 192.168.203.123@tcp [ 647.439754] Key type .llcrypt unregistered [ 647.446803] Key type ._llcrypt unregistered [ 666.293624] alg: No test for adler32 (adler32-zlib) [ 667.044488] Key type ._llcrypt registered [ 667.046348] Key type .llcrypt registered [ 667.162961] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 668.058261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 668.416682] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 668.622667] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 668.627653] LNet: Accept secure, port 988 [ 670.296285] Key type lgssc registered [ 671.291951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 678.030866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.324236] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.369859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 682.985166] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 684.515353] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.625126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.636553] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.880855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 693.263782] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 699.365058] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 703.817319] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 708.899205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 709.201653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 710.262546] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 712.226661] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 717.981663] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 729.967674] Lustre: lustre-OST0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 731.122381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.123229] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 731.131565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 731.134087] Lustre: Skipped 1 previous similar message [ 731.142757] LustreError: Skipped 1 previous similar message [ 732.741462] Lustre: server umount lustre-OST0000 complete [ 751.586157] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 751.602814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.635851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 751.637763] Lustre: Skipped 1 previous similar message [ 752.612724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.696377] Lustre: server umount lustre-MDT0000 complete [ 757.729641] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 757.749964] LustreError: Skipped 1 previous similar message [ 758.813707] LustreError: 34508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314341 with bad export cookie 5394368637810403337 [ 758.818375] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 758.825783] LustreError: 34508:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 765.202139] Lustre: server umount lustre-MDT0001 complete [ 774.404761] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 776.412829] Key type lgssc unregistered [ 776.611151] LNet: 38356:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.620990] LNet: Removed LNI 192.168.203.123@tcp [ 777.115021] Key type .llcrypt unregistered [ 777.117699] Key type ._llcrypt unregistered [ 793.148466] alg: No test for adler32 (adler32-zlib) [ 793.900335] Key type ._llcrypt registered [ 793.901840] Key type .llcrypt registered [ 793.992889] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 794.978755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 795.270760] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 795.449476] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 795.454420] LNet: Accept secure, port 988 [ 797.096471] Key type lgssc registered [ 797.933580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 805.581627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.001374] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.044956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.111307] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 812.520204] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 817.642392] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.754211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.254057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 822.724192] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 828.592420] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 832.356122] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 836.666933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 836.884960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 840.118736] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 845.738991] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 857.327389] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 864.775855] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 867.024886] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 873.454368] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 873.465691] LNetError: 39121:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 873.469378] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 878.561323] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-2410h00m59s [ 879.865639] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 882.029692] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 883.171167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.172162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 883.176903] Lustre: Skipped 2 previous similar messages [ 883.181606] Lustre: Skipped 1 previous similar message [ 883.188106] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 883.191981] LNetError: 39118:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 883.197471] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 887.139127] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 887.147135] Lustre: Skipped 1 previous similar message [ 887.429182] Lustre: server umount lustre-MDT0000 complete [ 888.296056] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.306039] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 888.320065] LustreError: Skipped 2 previous similar messages [ 888.327895] LNetError: 39119:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 888.327937] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 890.482236] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 893.409813] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.413630] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 893.419029] LustreError: Skipped 3 previous similar messages [ 893.424735] LNetError: 39120:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 893.433166] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 895.328136] Lustre: 39131:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314470/real 1761314470] req@00000000b0fb3a25 x1846871984196672/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314477 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 895.364146] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 896.003621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.590250] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xd9408c4e2cb01beb to 0xd9408c4e2cb02052 [ 900.603882] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 900.876794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.930060] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:193 [ 904.297857] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 906.224355] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 906.239104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 906.242863] Lustre: Skipped 1 previous similar message [ 907.143162] Lustre: DEBUG MARKER: Wait for df (28865) ... [ 908.278715] Lustre: DEBUG MARKER: done [ 910.819637] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 910.826063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.833834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.333966] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 916.450843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 916.457652] Lustre: Skipped 2 previous similar messages [ 916.642575] Lustre: server umount lustre-OST0000 complete [ 921.569542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.586715] Lustre: Skipped 2 previous similar messages [ 921.603112] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 921.613505] LNetError: 39121:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 921.619259] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 925.436784] Lustre: server umount lustre-MDT0000 complete [ 926.689106] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.697338] LustreError: Skipped 8 previous similar messages [ 928.307668] LustreError: 42539:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314510 with bad export cookie 15654666572124921938 [ 928.308253] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 928.315349] LustreError: 42539:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 928.581339] Lustre: server umount lustre-MDT0001 complete [ 938.277477] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 940.589849] Key type lgssc unregistered [ 940.837604] LNet: 44955:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.850065] LNet: Removed LNI 192.168.203.123@tcp [ 941.566536] Key type .llcrypt unregistered [ 941.568754] Key type ._llcrypt unregistered [ 968.876591] alg: No test for adler32 (adler32-zlib) [ 969.628424] Key type ._llcrypt registered [ 969.629962] Key type .llcrypt registered [ 969.734638] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 982.619385] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 983.677749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.959234] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 984.181544] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 984.183893] LNet: Accept secure, port 988 [ 985.800124] Key type lgssc registered [ 986.660911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 994.833375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1000.975627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1006.036051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1011.283914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1016.018799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1016.079099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.245894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1017.272958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1017.329319] Lustre: lustre-MDT0000: new disk, initializing [ 1017.445232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.483353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1020.554404] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1028.403185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1028.447491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.528439] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1028.567987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1028.571067] Lustre: Skipped 1 previous similar message [ 1028.644249] Lustre: lustre-MDT0001: new disk, initializing [ 1028.716463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1028.754114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1028.761562] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1031.681327] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1039.463772] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1043.730041] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1048.653726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1048.713504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1048.718890] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1048.929709] Lustre: lustre-OST0000: new disk, initializing [ 1048.938202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1049.047859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.751470] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1058.380812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1058.393328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1060.544427] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1064.309774] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1064.503610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1067.732304] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1067.890462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1074.150730] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.150869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.151610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1074.171876] Lustre: Skipped 1 previous similar message [ 1075.397262] Lustre: server umount lustre-OST0000 complete [ 1079.266531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.288309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.296582] Lustre: Skipped 2 previous similar messages [ 1084.385778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.389402] Lustre: Skipped 1 previous similar message [ 1089.505759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.512402] Lustre: Skipped 1 previous similar message [ 1092.064148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1092.157653] Lustre: server umount lustre-MDT0000 complete [ 1094.864027] LustreError: 49057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314677 with bad export cookie 530219680728036457 [ 1094.865686] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.870665] LustreError: 49057:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1095.089781] Lustre: server umount lustre-MDT0001 complete [ 1101.437229] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 10:04:43 (1761314683) [ 1106.769721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.186515] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1107.272359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.212543] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1112.546521] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.665642] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.837588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.135301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1121.860087] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1127.746910] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1132.208097] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1136.988129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1137.219089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.787789] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1146.767950] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1155.625326] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-2410h05m36s [ 1157.030953] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1162.489468] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1164.115557] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1164.208521] LustreError: 53607:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1165.296462] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 1165.302604] LNetError: 47276:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1165.308575] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1184.665751] Lustre: lustre-MDT0000: Client 12c366cd-ad6d-43a1-b43b-43ea9ad95b2e (at 192.168.203.23@tcp) reconnecting [ 1204.224237] LustreError: 53607:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 136 awake [ 1205.839277] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1207.761813] Lustre: DEBUG MARKER: done [ 1214.945164] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1214.949144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.957441] LustreError: Skipped 1 previous similar message [ 1214.963211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1214.964343] Lustre: Skipped 2 previous similar messages [ 1214.992880] Lustre: Skipped 3 previous similar messages [ 1218.006500] Lustre: server umount lustre-OST0000 complete [ 1225.192650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.213183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.223237] Lustre: Skipped 1 previous similar message [ 1225.238692] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 1225.250359] LNetError: 47273:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1225.260323] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1227.473166] Lustre: server umount lustre-MDT0000 complete [ 1230.306779] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.313823] LustreError: Skipped 1 previous similar message [ 1230.324198] LNetError: 120-3: Refusing connection from 192.168.203.123 for 127.0.0.2@tcp: No matching NI [ 1230.338918] LNetError: 47274:0:(socklnd_cb.c:1771:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1230.341556] LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1230.506112] LustreError: 56403:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314812 with bad export cookie 530219680728037311 [ 1230.508386] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.519441] LustreError: 56403:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1240.009710] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1242.195146] Key type lgssc unregistered [ 1242.388558] LNet: 58020:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.404614] LNet: Removed LNI 192.168.203.123@tcp [ 1243.118579] Key type .llcrypt unregistered [ 1243.120403] Key type ._llcrypt unregistered [ 1270.423935] alg: No test for adler32 (adler32-zlib) [ 1271.175127] Key type ._llcrypt registered [ 1271.176850] Key type .llcrypt registered [ 1271.273254] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1280.798924] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1281.493130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1281.740354] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 1281.884559] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1281.886955] LNet: Accept secure, port 988 [ 1283.512800] Key type lgssc registered [ 1284.231416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1291.213332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1297.708703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1302.914731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1308.004483] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1313.369599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1313.414865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.605931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1314.633353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1314.685653] Lustre: lustre-MDT0000: new disk, initializing [ 1314.778464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.797996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1317.629731] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1325.854760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1325.895937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.975599] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1326.025125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1326.027460] Lustre: Skipped 1 previous similar message [ 1326.083819] Lustre: lustre-MDT0001: new disk, initializing [ 1326.143227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1326.170644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1326.175668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1328.913124] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1335.590975] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1340.211417] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1345.295688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1345.344172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1345.349194] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1345.501667] Lustre: lustre-OST0000: new disk, initializing [ 1345.506861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1345.566832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.707758] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1353.272796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1353.279788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1356.009074] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1359.229791] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1359.375433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1362.554924] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1362.701664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1369.057729] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1369.058185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.059348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1369.062857] LustreError: Skipped 1 previous similar message [ 1370.253403] Lustre: server umount lustre-OST0000 complete [ 1374.179540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.181353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.187102] Lustre: Skipped 2 previous similar messages [ 1374.191691] Lustre: Skipped 1 previous similar message [ 1379.049260] Lustre: server umount lustre-MDT0000 complete [ 1379.297073] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1379.308044] LustreError: Skipped 1 previous similar message [ 1381.297425] LustreError: 62112:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314963 with bad export cookie 3245567526303486243 [ 1381.298590] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1381.309648] LustreError: 62112:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1381.504558] Lustre: server umount lustre-MDT0001 complete [ 1387.554270] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 10:09:29 (1761314969) [ 1388.349229] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1389.351987] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 10:09:31 (1761314971) [ 1395.835220] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1398.008753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.256375] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.300473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.948685] Lustre: Failing over lustre-MDT0000 [ 1398.952301] LustreError: 67132:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1398.957349] LustreError: 67132:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1398.962874] LustreError: 67132:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 1405.409347] Lustre: 67183:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314981/real 1761314981] req@000000005c7a54ac x1846872494853504/t0(0) o251->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314987 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1405.486473] Lustre: server umount lustre-MDT0000 complete [ 1408.955795] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:09:50 (1761314990) [ 1412.519927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.773333] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.801439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.574654] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1418.222223] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.140324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.386744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1424.031588] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1429.222728] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1433.269466] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1438.668985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1438.941785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.037558] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1447.993788] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1456.281883] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1462.240886] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1462.250979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.270630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1463.570915] Lustre: server umount lustre-MDT0000 complete [ 1466.385430] LustreError: 67731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315048 with bad export cookie 3245567526303487370 [ 1466.393773] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1469.725582] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1472.669451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1475.041462] Lustre: 60340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315050/real 1761315050] req@00000000095ea192 x1846872494870208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315057 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1475.064712] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.081962] Lustre: Skipped 2 previous similar messages [ 1479.137728] Lustre: 60339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315054/real 1761315054] req@00000000d918d3c2 x1846872494870272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315061 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1479.684835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.309003] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0x2d0a92782bdcd58a to 0x2d0a92782bdcdd16 [ 1483.329422] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 1483.736673] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1483.857372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.729574] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1491.942247] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.953327] LustreError: Skipped 2 previous similar messages [ 1495.032530] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1495.078864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1495.484550] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:65 [ 1498.796211] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1500.647920] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1504.976452] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1509.252290] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1516.005329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1516.006435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1516.009766] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1516.020694] Lustre: Skipped 2 previous similar messages [ 1516.039295] Lustre: Skipped 4 previous similar messages [ 1516.169432] Lustre: server umount lustre-MDT0000 complete [ 1516.170962] Lustre: Skipped 1 previous similar message [ 1519.010914] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1519.025713] LustreError: Skipped 6 previous similar messages [ 1519.306362] LustreError: 67731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315101 with bad export cookie 3245567526303489302 [ 1519.311039] LustreError: 67731:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1519.313510] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1524.371357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1530.995945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.360316] Lustre: 60341:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315106/real 1761315106] req@000000004be05226 x1846872494884032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315113 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1531.376053] Lustre: 60341:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1531.382497] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1536.480745] Lustre: 60338:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315111/real 1761315111] req@00000000e9595832 x1846872494884224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315118 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1536.501650] Lustre: 60338:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1536.546553] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0x2d0a92782bdcdd16 to 0x2d0a92782bdce328 [ 1536.555613] Lustre: MGC192.168.203.123@tcp: Connection restored to (at 0@lo) [ 1536.884468] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 1539.781913] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1545.201234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.437878] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:97 [ 1548.080721] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1550.829206] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1550.834475] Lustre: Skipped 1 previous similar message [ 1553.746353] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1558.164119] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1566.178595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.180604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.183717] Lustre: Skipped 2 previous similar messages [ 1566.186705] Lustre: Skipped 1 previous similar message [ 1566.188921] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1568.365992] Lustre: server umount lustre-MDT0000 complete [ 1568.369548] Lustre: Skipped 1 previous similar message [ 1570.694558] LustreError: 70705:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315153 with bad export cookie 3245567526303490856 [ 1570.696314] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1570.704186] LustreError: 70705:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1574.978522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1577.245956] Lustre: DEBUG MARKER: files compared the same [ 1578.383814] Lustre: 60339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315153/real 1761315153] req@000000003c7f587c x1846872494895616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315160 ref 1 fl Rpc:EXNQr/0/ffffffff rc -5/-1 job:'' [ 1578.394828] Lustre: 60339:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1594.363279] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1596.242642] Key type lgssc unregistered [ 1596.457783] LNet: 77788:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.464206] LNet: Removed LNI 192.168.203.123@tcp [ 1597.081420] Key type .llcrypt unregistered [ 1597.084188] Key type ._llcrypt unregistered [ 1611.911620] alg: No test for adler32 (adler32-zlib) [ 1612.664392] Key type ._llcrypt registered [ 1612.666266] Key type .llcrypt registered [ 1612.732886] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1613.275717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1613.647930] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 1613.805773] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1613.809662] LNet: Accept secure, port 988 [ 1615.440479] Key type lgssc registered [ 1616.484842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1622.595216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.968322] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1624.024548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1628.062320] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all 8 [ 1629.153798] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.278945] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.693429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1635.010620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1638.547458] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all 8 [ 1643.967318] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1647.076255] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1650.913744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1651.089779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.193814] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1654.428818] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug +malloc all 8 [ 1659.770392] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1665.445329] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:129 [ 1669.601720] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1669.605990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.613326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1670.626029] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1670.627661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1670.628399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1670.647898] Lustre: Skipped 1 previous similar message [ 1674.942901] Lustre: server umount lustre-OST0000 complete [ 1680.866956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.868041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1680.872441] Lustre: Skipped 1 previous similar message [ 1683.868148] Lustre: server umount lustre-MDT0000 complete [ 1685.990191] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.999052] LustreError: Skipped 1 previous similar message [ 1686.968111] LustreError: 79231:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315269 with bad export cookie 17543566269901845293 [ 1686.984131] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.206779] Lustre: server umount lustre-MDT0001 complete [ 1694.891080] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1696.955913] Key type lgssc unregistered [ 1697.171406] LNet: 82902:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.178264] LNet: Removed LNI 192.168.203.123@tcp [ 1697.779685] Key type .llcrypt unregistered [ 1697.781594] Key type ._llcrypt unregistered [ 1715.046731] alg: No test for adler32 (adler32-zlib) [ 1715.806061] Key type ._llcrypt registered [ 1715.810491] Key type .llcrypt registered [ 1715.931795] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1716.570892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.971897] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 1717.144970] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1717.150871] LNet: Accept secure, port 988 [ 1718.832120] Key type lgssc registered [ 1719.708279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1725.669259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1742.241239] LustreError: 84336:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 1: rc = -5 [ 1773.984282] LustreError: 84336:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 1803.744195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.814327] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1813.828731] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1820.311572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.564575] Lustre: *** cfs_fail_loc=706, val=0*** [ 1826.566901] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.625546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.696296] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:161 [ 1829.603183] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1831.906080] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.028340] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.297070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.593993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1837.643259] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:161 [ 1841.174787] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1848.221302] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1852.222207] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1857.506666] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1857.521208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.546933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.020277] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1858.024670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.043101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.048461] Lustre: Skipped 2 previous similar messages [ 1864.163423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.166170] Lustre: Skipped 1 previous similar message [ 1868.256319] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1868.369609] Lustre: server umount lustre-OST0000 complete [ 1873.380027] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1873.386874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.398080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.520932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.527820] Lustre: Skipped 3 previous similar messages [ 1885.153087] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1885.257418] Lustre: server umount lustre-MDT0000 complete [ 1888.325795] LustreError: 87272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315470 with bad export cookie 8575096870446298244 [ 1888.329418] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.342367] LustreError: 87272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1888.581355] Lustre: server umount lustre-MDT0001 complete [ 1897.447955] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1900.123690] Key type lgssc unregistered [ 1900.337959] LNet: 88086:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.345642] LNet: Removed LNI 192.168.203.123@tcp [ 1901.158959] Key type .llcrypt unregistered [ 1901.160674] Key type ._llcrypt unregistered [ 1919.485892] alg: No test for adler32 (adler32-zlib) [ 1920.238688] Key type ._llcrypt registered [ 1920.240969] Key type .llcrypt registered [ 1920.314848] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1921.036185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1921.337260] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 1921.488818] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1921.491463] LNet: Accept secure, port 988 [ 1923.137555] Key type lgssc registered [ 1923.817769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1928.750555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.096691] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1937.358911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.614012] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.711317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1940.758723] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1943.014606] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.020280] LustreError: Skipped 2 previous similar messages [ 1947.000107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1947.221135] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.233820] LustreError: Skipped 2 previous similar messages [ 1947.252454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.118432] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1952.740589] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.754968] LustreError: Skipped 2 previous similar messages [ 1953.781651] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:43 to 0x280000400:193 [ 1957.640064] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1957.652804] LustreError: Skipped 5 previous similar messages [ 1957.816201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.864776] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:193 [ 1960.650533] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 1973.218675] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1973.221343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.246401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.249829] Lustre: Skipped 1 previous similar message [ 1976.732045] Lustre: server umount lustre-OST0000 complete [ 1983.463897] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1983.465419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.480889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.497878] Lustre: Skipped 1 previous similar message [ 1983.502461] Lustre: Skipped 1 previous similar message [ 1986.337782] Lustre: server umount lustre-MDT0000 complete [ 1988.578403] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.585688] LustreError: Skipped 1 previous similar message [ 1989.842992] Lustre: server umount lustre-MDT0001 complete [ 1999.328285] Lustre: 92950:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315575/real 1761315575] req@00000000ebccc902 x1846873164886336/t0(0) o251->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315581 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1999.383290] Lustre: server umount MGS complete [ 2006.844950] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:19:48 (1761315588) [ 2026.998636] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2033.420578] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2042.844696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2047.958748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2052.418685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2056.956120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2062.342786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2062.400188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.642402] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2072.869938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2072.910743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.046809] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2073.088911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2073.140939] Lustre: lustre-MDT0001: new disk, initializing [ 2073.203130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2073.222464] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2073.229314] LustreError: 97649:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osd get update log failed: rc = -115 [ 2076.694534] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2085.245248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2085.295575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2085.303591] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2085.484063] Lustre: lustre-OST0000: new disk, initializing [ 2085.487525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2085.539664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.357242] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2096.521815] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2096.673860] Lustre: 99088:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@0000000012778ce6 x1846873164891008/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761315691 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2102.530625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2102.581249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2102.658280] Lustre: lustre-MDT0000: new disk, initializing [ 2102.769116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.806281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2103.842260] Lustre: 99499:0:(client.c:1485:after_reply()) @@@ resending request on EINPROGRESS req@00000000bb2cfc10 x1846873164894272/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1761315698 ref 2 fl Rpc:RQU/2/0 rc 0/-115 job:'' [ 2105.917152] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2107.302027] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2108.974646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2108.983408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2110.957127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2110.963025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2131.425043] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2131.429690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.436300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2131.438577] Lustre: Skipped 1 previous similar message [ 2133.474894] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2133.485186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.496463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2135.750494] Lustre: server umount lustre-OST0000 complete [ 2139.105846] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2139.110722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.111133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.111136] Lustre: Skipped 1 previous similar message [ 2144.643865] Lustre: server umount lustre-MDT0000 complete [ 2163.680157] Lustre: 100694:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315740/real 1761315740] req@000000002799398d x1846873164903680/t0(0) o251->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315746 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2163.730644] Lustre: server umount MGS complete [ 2163.731967] Lustre: Skipped 1 previous similar message [ 2171.328669] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:22:33 (1761315753) [ 2184.782257] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2187.386653] Key type lgssc unregistered [ 2187.674189] LNet: 102099:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.687611] LNet: Removed LNI 192.168.203.123@tcp [ 2188.329949] Key type .llcrypt unregistered [ 2188.331525] Key type ._llcrypt unregistered [ 2201.290203] alg: No test for adler32 (adler32-zlib) [ 2202.043562] Key type ._llcrypt registered [ 2202.045547] Key type .llcrypt registered [ 2202.111737] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2202.881044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2203.139579] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2203.315461] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2203.319957] LNet: Accept secure, port 988 [ 2205.003339] Key type lgssc registered [ 2205.607983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2211.976678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.338753] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2213.384780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.655538] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2218.470817] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.606890] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.766788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.986523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2228.182552] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2234.704235] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2239.588877] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2245.678416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2246.006129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.501615] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2254.985879] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2263.404786] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 2273.764515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.767131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2273.786962] Lustre: Skipped 1 previous similar message [ 2277.492645] Lustre: server umount lustre-OST0000 complete [ 2281.441038] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2281.451810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.472862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2281.480790] Lustre: Skipped 1 previous similar message [ 2284.001723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.011159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.122254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.125177] Lustre: Skipped 1 previous similar message [ 2295.264466] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2295.268668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.287545] Lustre: Skipped 3 previous similar messages [ 2295.405769] Lustre: server umount lustre-MDT0000 complete [ 2299.396993] LustreError: 103349:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315881 with bad export cookie 9278091555470662463 [ 2299.399777] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.416349] LustreError: 103349:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2299.975054] Lustre: server umount lustre-MDT0001 complete [ 2308.612303] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2310.341616] Key type lgssc unregistered [ 2310.510062] LNet: 107046:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.517169] LNet: Removed LNI 192.168.203.123@tcp [ 2311.054521] Key type .llcrypt unregistered [ 2311.056978] Key type ._llcrypt unregistered [ 2322.576922] alg: No test for adler32 (adler32-zlib) [ 2323.329416] Key type ._llcrypt registered [ 2323.332112] Key type .llcrypt registered [ 2323.410327] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2324.076053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2324.329694] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2324.451302] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2324.456765] LNet: Accept secure, port 988 [ 2326.064148] Key type lgssc registered [ 2326.987753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2332.228107] LustreError: 108175:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2332.231663] LustreError: 108173:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2332.236691] LustreError: 108175:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4996 [ 2332.243351] LustreError: 108173:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2332.252366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.268534] LustreError: 108173:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2333.604724] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.656451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.786139] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.033903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.221694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2342.242285] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2346.272907] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing lsmod [ 2348.257359] LustreError: 109481:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 716 sleeping [ 2348.264538] LustreError: 109482:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 716 waking [ 2348.273657] LustreError: 109481:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 716 awake: rc=4990 [ 2348.281492] LustreError: 109481:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2348.289436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2348.293098] LustreError: 109481:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2348.616469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.680278] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 2353.764367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.770973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2353.775597] Lustre: Skipped 1 previous similar message [ 2353.796470] Lustre: Skipped 2 previous similar messages [ 2356.749757] Lustre: server umount lustre-MDT0000 complete [ 2359.267759] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.280074] LustreError: Skipped 2 previous similar messages [ 2359.796202] LustreError: 108191:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315942 with bad export cookie 14678537769947154300 [ 2359.800429] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2359.806210] LustreError: 108191:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2360.099191] Lustre: server umount lustre-MDT0001 complete [ 2369.504176] Lustre: 110075:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315945/real 1761315945] req@000000000e357326 x1846873587459648/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315951 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2369.730537] Lustre: server umount lustre-OST0000 complete [ 2377.021610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.457638] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.546688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.798828] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2382.821769] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.410875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.701355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.072357] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2397.692927] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2402.525085] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2407.984700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2408.280863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.335833] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 2411.953688] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2418.535789] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2423.777876] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2423.782545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.792156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2423.799299] Lustre: Skipped 2 previous similar messages [ 2425.078757] Lustre: server umount lustre-OST0000 complete [ 2428.896878] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2428.898657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.903390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2428.917563] Lustre: Skipped 1 previous similar message [ 2428.923820] Lustre: Skipped 1 previous similar message [ 2434.019533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2434.031795] Lustre: Skipped 2 previous similar messages [ 2434.658435] Lustre: server umount lustre-MDT0000 complete [ 2437.762500] LustreError: 110531:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316020 with bad export cookie 14678537769947155231 [ 2437.763670] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2437.766933] LustreError: 110531:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2448.120663] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2450.192179] Key type lgssc unregistered [ 2450.394327] LNet: 114204:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.404229] LNet: Removed LNI 192.168.203.123@tcp [ 2451.134134] Key type .llcrypt unregistered [ 2451.136451] Key type ._llcrypt unregistered [ 2471.067954] alg: No test for adler32 (adler32-zlib) [ 2471.819585] Key type ._llcrypt registered [ 2471.820961] Key type .llcrypt registered [ 2471.920217] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2472.537433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2473.053323] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2473.279449] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2473.283390] LNet: Accept secure, port 988 [ 2474.920164] Key type lgssc registered [ 2475.614840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2480.623306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.950745] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.999507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.251248] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2487.265951] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.387764] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.865139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.255049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2497.070633] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2503.344980] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2508.165285] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2514.166894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2514.491911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.926762] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2523.799540] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2531.893208] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 2537.692852] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2540.144377] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 2545.491768] Lustre: server umount lustre-OST0000 complete [ 2549.729401] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2549.738869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.745664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2552.305392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2552.310166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.824185] Lustre: 118640:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316130/real 1761316130] req@000000001d96f744 x1846873743705088/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761316136 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2553.826171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.854990] Lustre: 118640:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2553.873421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.884841] Lustre: Skipped 2 previous similar messages [ 2553.999601] Lustre: server umount lustre-MDT0000 complete [ 2555.362095] Lustre: 114979:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316130/real 1761316130] req@00000000add83cbe x1846873743704576/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761316137 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 2556.762615] LustreError: 115648:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316139 with bad export cookie 298010436296035652 [ 2556.764909] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2556.776913] LustreError: 115648:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2557.045513] Lustre: server umount lustre-MDT0001 complete [ 2565.109367] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2566.763465] Key type lgssc unregistered [ 2566.919746] LNet: 119443:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2566.924099] LNet: Removed LNI 192.168.203.123@tcp [ 2567.371319] Key type .llcrypt unregistered [ 2567.372956] Key type ._llcrypt unregistered [ 2577.266060] alg: No test for adler32 (adler32-zlib) [ 2578.018431] Key type ._llcrypt registered [ 2578.020063] Key type .llcrypt registered [ 2578.120741] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2578.523897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2578.826220] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2578.950518] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2578.953984] LNet: Accept secure, port 988 [ 2580.576199] Key type lgssc registered [ 2581.198565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2586.817153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.124588] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.162776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.322358] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2593.252431] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.214857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.369847] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2598.527762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2601.500557] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2607.047517] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2611.559324] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2616.436507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.703598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.709448] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 2619.370340] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2623.893363] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2633.187387] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2633.194902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.204036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2633.209670] Lustre: Skipped 1 previous similar message [ 2638.307161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2638.309704] Lustre: Skipped 1 previous similar message [ 2638.827458] Lustre: server umount lustre-OST0000 complete [ 2643.427042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.435452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2643.441193] Lustre: Skipped 2 previous similar messages [ 2643.450411] Lustre: Skipped 1 previous similar message [ 2648.550129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2648.556900] Lustre: Skipped 1 previous similar message [ 2656.230209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2656.320304] Lustre: server umount lustre-MDT0000 complete [ 2658.785145] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.791428] LustreError: Skipped 1 previous similar message [ 2659.253973] LustreError: 120689:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316241 with bad export cookie 11682656308566410799 [ 2659.257657] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2659.263325] LustreError: 120689:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2659.589549] Lustre: server umount lustre-MDT0001 complete [ 2668.265965] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2669.976701] Key type lgssc unregistered [ 2670.184476] LNet: 124364:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2670.194338] LNet: Removed LNI 192.168.203.123@tcp [ 2670.867591] Key type .llcrypt unregistered [ 2670.869258] Key type ._llcrypt unregistered [ 2688.681795] alg: No test for adler32 (adler32-zlib) [ 2689.435538] Key type ._llcrypt registered [ 2689.437613] Key type .llcrypt registered [ 2689.563754] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2690.202083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2690.583978] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2690.761892] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2690.764272] LNet: Accept secure, port 988 [ 2692.376261] Key type lgssc registered [ 2693.099321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2698.061392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.372733] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.439518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.224476] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2704.869442] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.225632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.485905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2711.260320] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2715.481707] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2718.543445] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2722.439677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2722.693590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.107864] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2729.839733] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 2732.849582] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2739.191298] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2742.393813] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2742.399626] Lustre: Skipped 2 previous similar messages [ 2746.029388] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 2746.037134] Lustre: Skipped 2 previous similar messages [ 2746.927716] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2746.965181] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2755.066366] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 2755.070870] Lustre: Skipped 2 previous similar messages [ 2760.788225] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp [ 2764.771310] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2764.777926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2764.790799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.798911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.809428] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2765.813449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2769.834595] Lustre: server umount lustre-OST0000 complete [ 2775.008878] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2775.017668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2775.029815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2775.032887] Lustre: Skipped 1 previous similar message [ 2778.696821] Lustre: server umount lustre-MDT0000 complete [ 2781.154588] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.162721] LustreError: Skipped 1 previous similar message [ 2781.540453] LustreError: 125806:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316363 with bad export cookie 584261645346318760 [ 2781.542919] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2781.560994] LustreError: 125806:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2781.861746] Lustre: server umount lustre-MDT0001 complete [ 2791.803489] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2794.171313] Key type lgssc unregistered [ 2794.379607] LNet: 130479:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2794.386798] LNet: Removed LNI 192.168.203.123@tcp [ 2795.089356] Key type .llcrypt unregistered [ 2795.093555] Key type ._llcrypt unregistered [ 2815.480436] alg: No test for adler32 (adler32-zlib) [ 2816.232439] Key type ._llcrypt registered [ 2816.236312] Key type .llcrypt registered [ 2816.347911] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2817.124226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2817.378798] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2817.537301] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2817.539900] LNet: Accept secure, port 988 [ 2819.193196] Key type lgssc registered [ 2819.920632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2825.607180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.938399] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.979234] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2827.013683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.017446] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp [ 2830.216608] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2832.356957] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.489810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.647818] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2836.660279] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.23@tcp 192.168.0.[2,10]@tcp [ 2836.688075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2839.648912] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2845.210663] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2849.678599] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2854.486764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2854.729123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.821755] Lustre: lustre-OST0000: deleting orphan objects from 0x0:199 to 0x0:225 [ 2857.538775] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2863.604722] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2876.898885] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2876.903833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.913738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2877.416292] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2877.423765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.424978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2877.454241] Lustre: Skipped 1 previous similar message [ 2880.442617] Lustre: server umount lustre-OST0000 complete [ 2887.649271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.650227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2887.656478] Lustre: Skipped 1 previous similar message [ 2889.705428] Lustre: server umount lustre-MDT0000 complete [ 2892.048355] LustreError: 132065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316474 with bad export cookie 5774098871467063596 [ 2892.051721] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2892.057451] LustreError: 132065:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2892.362186] Lustre: server umount lustre-MDT0001 complete [ 2902.695263] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2905.998773] Key type lgssc unregistered [ 2906.389572] LNet: 135795:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2906.412094] LNet: Removed LNI 192.168.203.123@tcp [ 2907.330914] Key type .llcrypt unregistered [ 2907.343481] Key type ._llcrypt unregistered [ 2931.140732] alg: No test for adler32 (adler32-zlib) [ 2931.894521] Key type ._llcrypt registered [ 2931.896211] Key type .llcrypt registered [ 2931.960220] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2941.965805] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2942.870652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2943.083949] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 2943.201547] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2943.204669] LNet: Accept secure, port 988 [ 2944.840129] Key type lgssc registered [ 2945.636727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2952.453526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2958.027328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2963.356059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2969.564576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2975.478863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2975.526288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.768345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2976.813408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2976.927475] Lustre: lustre-MDT0000: new disk, initializing [ 2977.056123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2977.073151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2980.704896] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 2990.089367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2990.167723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.276130] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2990.328647] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.353685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2990.358228] Lustre: Skipped 1 previous similar message [ 2990.419443] Lustre: lustre-MDT0001: new disk, initializing [ 2990.514025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2990.553803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2990.564551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2993.524617] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3000.214881] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3003.907328] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3008.567748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3008.614411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3008.619920] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3008.770746] Lustre: lustre-OST0000: new disk, initializing [ 3008.777206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3008.843834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3011.556788] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3017.794568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3017.802738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3019.319329] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3022.486948] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3022.661244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3025.452865] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3025.614605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3028.450967] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3028.453074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.454465] LustreError: Skipped 1 previous similar message [ 3028.462549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3028.469067] Lustre: Skipped 1 previous similar message [ 3033.571970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3033.574927] Lustre: Skipped 1 previous similar message [ 3038.690964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3038.693948] Lustre: Skipped 1 previous similar message [ 3041.248844] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3041.372995] Lustre: server umount lustre-OST0000 complete [ 3043.810336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3043.816124] Lustre: Skipped 1 previous similar message [ 3043.816720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3043.820936] Lustre: Skipped 1 previous similar message [ 3048.929866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3048.934619] Lustre: Skipped 1 previous similar message [ 3049.818485] Lustre: server umount lustre-MDT0000 complete [ 3051.940482] LustreError: 140167:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316634 with bad export cookie 1676392690956735748 [ 3051.944359] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3051.946477] LustreError: 140167:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3052.154070] Lustre: server umount lustre-MDT0001 complete [ 3057.280549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.619597] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.687606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.558498] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3062.756024] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.202701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.473464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3069.172404] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3074.681932] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3078.688279] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3083.157595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3083.379885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3086.097445] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3091.237444] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3094.112716] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3094.251217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3096.895664] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3097.038121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3102.253544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3102.299082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3102.307273] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3102.415048] Lustre: lustre-OST0001: new disk, initializing [ 3102.417494] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3102.463921] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3105.133696] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3111.374355] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 3111.488530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3111.494149] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3112.626801] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3115.707931] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 3115.846913] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3120.083314] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff989342e12800.ost_server_uuid 40 [ 3121.082471] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff989342e12800.ost_server_uuid in FULL state after 0 sec [ 3127.273807] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.274500] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3127.275433] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3127.291340] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3127.303569] Lustre: Skipped 1 previous similar message [ 3131.414122] Lustre: server umount lustre-OST0001 complete [ 3132.394281] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.405756] LustreError: Skipped 1 previous similar message [ 3134.946637] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3134.953108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.959930] Lustre: Skipped 1 previous similar message [ 3137.507792] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.522112] LustreError: Skipped 1 previous similar message [ 3140.561275] Lustre: server umount lustre-OST0000 complete [ 3147.746604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.752261] Lustre: Skipped 1 previous similar message [ 3147.756946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3147.760390] Lustre: Skipped 4 previous similar messages [ 3149.448191] Lustre: server umount lustre-MDT0000 complete [ 3152.405252] LustreError: 143656:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316734 with bad export cookie 1676392690956736602 [ 3152.405695] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3152.411741] LustreError: 143656:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3169.402322] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3171.328888] Key type lgssc unregistered [ 3171.514045] LNet: 150330:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3171.519912] LNet: Removed LNI 192.168.203.123@tcp [ 3172.075282] Key type .llcrypt unregistered [ 3172.077358] Key type ._llcrypt unregistered [ 3197.614262] alg: No test for adler32 (adler32-zlib) [ 3198.368128] Key type ._llcrypt registered [ 3198.370669] Key type .llcrypt registered [ 3198.477532] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3209.519890] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3210.397562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3210.927228] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 3211.114187] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3211.124989] LNet: Accept secure, port 988 [ 3212.776159] Key type lgssc registered [ 3213.638809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3221.156026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3226.652291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3231.488700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3236.078096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3240.930092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3240.987941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.155358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3242.192813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3242.283394] Lustre: lustre-MDT0000: new disk, initializing [ 3242.370223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3242.382187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3245.329515] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3253.575518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3253.636228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.713961] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3253.759952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3253.763370] Lustre: Skipped 1 previous similar message [ 3253.824785] Lustre: lustre-MDT0001: new disk, initializing [ 3253.921277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3253.961920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3253.967151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3257.418406] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3266.039605] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3270.675354] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3276.855356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3276.917481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3276.921578] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3277.094424] Lustre: lustre-OST0000: new disk, initializing [ 3277.099948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3277.157736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.734422] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3285.024270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3285.030225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3288.890349] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3291.931614] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3292.067335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3294.458926] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3294.600419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3295.713689] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3295.719356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.728512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3300.321815] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3300.326395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.335345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3301.793825] Lustre: server umount lustre-OST0000 complete [ 3305.443626] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3305.449847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.464064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3305.468300] Lustre: Skipped 1 previous similar message [ 3310.581194] Lustre: server umount lustre-MDT0000 complete [ 3311.074726] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3313.577898] LustreError: 154421:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316896 with bad export cookie 14644533569591933374 [ 3313.584032] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3313.588079] LustreError: 154421:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3313.813993] Lustre: server umount lustre-MDT0001 complete [ 3320.218296] Lustre: DEBUG MARKER: == conf-sanity test 47: server restart does not make client loss lru_resize settings ========================================================== 10:41:41 (1761316901) [ 3333.198516] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3341.013647] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3349.058826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3354.335323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3358.989756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3363.720056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3368.507776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3368.569644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.721610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3368.760653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3368.815986] Lustre: lustre-MDT0000: new disk, initializing [ 3368.886217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.899195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3371.929760] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3379.318326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3379.364081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3379.417536] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3379.448738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3379.452350] Lustre: Skipped 1 previous similar message [ 3379.499788] Lustre: lustre-MDT0001: new disk, initializing [ 3379.565041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3379.586646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3379.592353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3382.721078] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3390.468124] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3395.546771] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3401.095249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3401.165279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.173986] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3401.347531] Lustre: lustre-OST0000: new disk, initializing [ 3401.357547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3401.438477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.729205] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3411.007887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3411.021552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3413.013196] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3417.669810] Lustre: Failing over lustre-OST0000 [ 3417.960543] Lustre: server umount lustre-OST0000 complete [ 3426.785944] Lustre: 152649:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317001/real 1761317001] req@0000000030a0beeb x1846874517568896/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1761317008 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3426.802308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.809403] Lustre: Skipped 1 previous similar message [ 3428.832507] Lustre: 152648:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317004/real 1761317004] req@000000006ac93c35 x1846874517569152/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761317011 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3433.953958] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317009/real 1761317009] req@00000000e16bc728 x1846874517569984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761317016 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3434.013551] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3437.410826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3437.823320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.844962] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3441.511900] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3443.558440] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3445.072425] Lustre: Failing over lustre-MDT0000 [ 3445.093353] LustreError: 152645:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3445.255901] Lustre: server umount lustre-MDT0000 complete [ 3445.988989] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.994564] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.015049] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 3446.016641] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 3446.020324] Lustre: Skipped 3 previous similar messages [ 3446.067517] LustreError: Skipped 3 previous similar messages [ 3448.692500] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.363720] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.374813] LustreError: Skipped 2 previous similar messages [ 3455.885913] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.897095] LustreError: Skipped 1 previous similar message [ 3458.465813] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317033/real 1761317033] req@000000007acdf34f x1846874517576640/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761317040 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3458.485758] Lustre: 152647:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3458.493702] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3463.435958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.656471] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xcb3bd5b104cf7d0d to 0xcb3bd5b104cf81e4 [ 3463.671559] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 3463.798206] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3463.886380] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3467.106723] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3467.397489] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3469.296458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3469.319948] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3469.354839] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3474.405802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.414831] Lustre: Skipped 1 previous similar message [ 3474.415784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3474.437735] Lustre: Skipped 2 previous similar messages [ 3476.332248] Lustre: server umount lustre-OST0000 complete [ 3479.522069] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3479.523151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3479.534497] Lustre: Skipped 1 previous similar message [ 3488.752147] LustreError: 161404:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317071 with bad export cookie 14644533569591935460 [ 3488.753561] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3488.763408] LustreError: 161404:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3499.068695] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3501.449893] Key type lgssc unregistered [ 3501.657628] LNet: 167258:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3501.672700] LNet: Removed LNI 192.168.203.123@tcp [ 3502.303331] Key type .llcrypt unregistered [ 3502.305105] Key type ._llcrypt unregistered [ 3520.918177] alg: No test for adler32 (adler32-zlib) [ 3521.669562] Key type ._llcrypt registered [ 3521.672681] Key type .llcrypt registered [ 3521.748921] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3522.350206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3522.610560] Lustre: Lustre: Build Version: 2.15.7_10_g2983877 [ 3522.746205] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3522.750981] LNet: Accept secure, port 988 [ 3524.426558] Key type lgssc registered [ 3525.226194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3530.856650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.198324] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3532.246227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.427633] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3537.378301] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.866884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3542.228991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3545.522898] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3550.856816] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3555.787564] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3561.107258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3561.376177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3562.413132] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 3564.254391] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 3570.456740] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3588.992537] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3596.160599] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3627.418674] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3636.603368] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3650.453522] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3658.619125] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3681.653709] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3723.646913] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3723.654212] Lustre: Skipped 1 previous similar message [ 3799.468436] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3799.492537] Lustre: Skipped 5 previous similar messages [ 3929.480414] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 3929.488087] Lustre: Skipped 2 previous similar messages [ 4224.399605] Lustre: lustre-MDT0000: Client 19b65da0-8bad-41f8-9b20-cdf56d976ec0 (at 192.168.203.23@tcp) reconnecting [ 4224.411214] Lustre: Skipped 5 previous similar messages [ 4389.737374] Lustre: Failing over lustre-MDT0000 [ 4389.858766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.862986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.874806] Lustre: Skipped 2 previous similar messages [ 4389.891497] Lustre: Skipped 2 previous similar messages [ 4389.929398] LustreError: 171563:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000007ee01726) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4390.002605] Lustre: server umount lustre-MDT0000 complete [ 4390.758522] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.981421] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.990099] LustreError: Skipped 2 previous similar messages [ 4397.933958] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.952411] LustreError: Skipped 1 previous similar message [ 4400.099524] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.118144] LustreError: Skipped 2 previous similar messages [ 4402.080395] Lustre: 168030:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317977/real 1761317977] req@000000006dc239bc x1846874845123776/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761317984 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4402.095944] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4407.274678] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.290179] LustreError: Skipped 7 previous similar messages [ 4407.899838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.303602] Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0x553624da0720cb83 to 0x553624da072337b2 [ 4408.316769] Lustre: MGC192.168.203.123@tcp: Connection restored to 192.168.203.123@tcp (at 0@lo) [ 4408.510480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4408.568564] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4409.189478] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4411.641757] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4413.934515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4413.950637] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4413.976323] Lustre: lustre-OST0000: deleting orphan objects from 0x0:68 to 0x0:97 [ 4418.606446] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4419.964987] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4439.522133] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4439.524354] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.533905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4439.541279] Lustre: Skipped 1 previous similar message [ 4441.585268] Lustre: server umount lustre-MDT0000 complete [ 4444.641609] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4444.656396] LustreError: Skipped 5 previous similar messages [ 4445.326084] LustreError: 169505:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318027 with bad export cookie 6140135660807468978 [ 4445.330267] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4445.331095] LustreError: 169505:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4449.764517] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.770648] Lustre: Skipped 1 previous similar message [ 4449.777313] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4449.779869] Lustre: Skipped 2 previous similar messages [ 4451.639567] Lustre: server umount lustre-MDT0001 complete [ 4460.557620] Lustre: server umount lustre-OST0000 complete [ 4467.125677] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4473.330090] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4480.775646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4485.350606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4489.653626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4493.844606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.725940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.777509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4498.953034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4498.968701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4499.021064] Lustre: lustre-MDT0000: new disk, initializing [ 4499.100790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.114077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4501.738775] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4508.007889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4508.056622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.111872] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4508.142373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4508.146059] Lustre: Skipped 1 previous similar message [ 4508.192333] Lustre: lustre-MDT0001: new disk, initializing [ 4508.269900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4508.301336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4508.308293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4510.949135] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4517.107095] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4520.877316] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4526.351821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4526.450186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4526.459939] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4526.785497] Lustre: lustre-OST0000: new disk, initializing [ 4526.788830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4526.859453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4530.694522] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4536.380555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4536.387857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4538.549652] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4541.936893] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 4542.184362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4545.591176] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 4545.727466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4552.162884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.163646] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4552.178941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4552.195074] Lustre: Skipped 2 previous similar messages [ 4553.758443] Lustre: server umount lustre-OST0000 complete [ 4557.282970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.287102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4557.296780] Lustre: Skipped 2 previous similar messages [ 4557.312891] Lustre: Skipped 1 previous similar message [ 4563.399684] Lustre: server umount lustre-MDT0000 complete [ 4566.597754] LustreError: 175692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318149 with bad export cookie 6140135660807469699 [ 4566.605070] LustreError: 166-1: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4566.613905] LustreError: 175692:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4572.898439] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:02:34 (1761318154) [ 4584.333873] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4589.409818] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4594.837765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4599.034056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4603.734471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4608.793281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.891869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.940499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4613.088758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4613.108848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4613.168795] Lustre: lustre-MDT0000: new disk, initializing [ 4613.257796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.279446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4616.281710] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4624.566345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4624.621335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.680965] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4624.709583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4624.713425] Lustre: Skipped 1 previous similar message [ 4624.788687] Lustre: lustre-MDT0001: new disk, initializing [ 4624.856037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4624.885467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4624.889929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4628.048742] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4636.141895] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4641.064732] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4646.298615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4646.372964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4646.377356] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4646.575197] Lustre: lustre-OST0000: new disk, initializing [ 4646.588955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4646.647186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.612690] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all 8 [ 4653.084874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4653.090682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4655.755442] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4678.464674] Lustre: DEBUG MARKER: conf-sanity test_49a: @@@@@@ FAIL: client mount failed