[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505318517 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003197] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008761] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.012126] LSM: Security Framework initializing [ 0.013067] Yama: becoming mindful. [ 0.014039] SELinux: Initializing. [ 0.015116] *** VALIDATE selinux *** [ 0.024527] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029134] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033132] *** VALIDATE tmpfs *** [ 0.035206] *** VALIDATE proc *** [ 0.036278] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.039199] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040132] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042029] Spectre V2 : User space: Vulnerable [ 0.043012] Speculative Store Bypass: Vulnerable [ 0.045648] debug: unmapping init [mem 0xffffffffb5059000-0xffffffffb5060fff] [ 0.047907] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048664] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049031] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051010] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053013] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.056341] rcu: Hierarchical SRCU implementation. [ 0.058548] smp: Bringing up secondary CPUs ... [ 0.059625] x86: Booting SMP configuration: [ 0.060031] .... node #0, CPUs: #1 #2 #3 [ 0.063698] smp: Brought up 1 node, 4 CPUs [ 0.065014] smpboot: Max logical packages: 1 [ 0.066021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.177000] node 0 deferred pages initialised in 108ms [ 0.181096] devtmpfs: initialized [ 0.183512] x86/mm: Memory block size: 128MB [ 0.187249] gcov: version magic: 0x41383552 [ 0.190338] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.193099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.197784] pinctrl core: initialized pinctrl subsystem [ 0.200172] [ 0.200675] ************************************************************* [ 0.203015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.206024] ** ** [ 0.207014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.210014] ** ** [ 0.211011] ** This means that this kernel is built to expose internal ** [ 0.214015] ** IOMMU data structures, which may compromise security on ** [ 0.217013] ** your system. ** [ 0.219014] ** ** [ 0.222014] ** If you see this message and you are not debugging the ** [ 0.224023] ** kernel, report this immediately to your vendor! ** [ 0.227012] ** ** [ 0.229013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.232014] ************************************************************* [ 0.235588] NET: Registered protocol family 16 [ 0.237544] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.239067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.241058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.244070] cpuidle: using governor menu [ 0.245947] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.249525] PCI: Using configuration type 1 for base access [ 0.250123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.259111] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.261052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.265177] cryptd: max_cpu_qlen set to 1000 [ 0.268261] ACPI: Added _OSI(Module Device) [ 0.270013] ACPI: Added _OSI(Processor Device) [ 0.271011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.273012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.277675] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.284589] ACPI: Interpreter enabled [ 0.286097] ACPI: PM: (supports S0 S3 S4 S5) [ 0.287017] ACPI: Using IOAPIC for interrupt routing [ 0.289133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.293395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.304275] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.306080] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.309024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.310076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.316450] acpiphp: Slot [2] registered [ 0.317095] acpiphp: Slot [5] registered [ 0.318108] acpiphp: Slot [6] registered [ 0.320121] acpiphp: Slot [7] registered [ 0.321103] acpiphp: Slot [8] registered [ 0.322090] acpiphp: Slot [9] registered [ 0.323099] acpiphp: Slot [10] registered [ 0.325102] acpiphp: Slot [3] registered [ 0.326121] acpiphp: Slot [4] registered [ 0.328105] acpiphp: Slot [11] registered [ 0.329101] acpiphp: Slot [12] registered [ 0.330121] acpiphp: Slot [13] registered [ 0.332083] acpiphp: Slot [14] registered [ 0.333097] acpiphp: Slot [15] registered [ 0.334129] acpiphp: Slot [16] registered [ 0.335098] acpiphp: Slot [17] registered [ 0.337090] acpiphp: Slot [18] registered [ 0.338088] acpiphp: Slot [19] registered [ 0.339155] acpiphp: Slot [20] registered [ 0.340088] acpiphp: Slot [21] registered [ 0.341097] acpiphp: Slot [22] registered [ 0.343115] acpiphp: Slot [23] registered [ 0.344098] acpiphp: Slot [24] registered [ 0.346115] acpiphp: Slot [25] registered [ 0.348104] acpiphp: Slot [26] registered [ 0.349176] acpiphp: Slot [27] registered [ 0.351172] acpiphp: Slot [28] registered [ 0.353091] acpiphp: Slot [29] registered [ 0.355137] acpiphp: Slot [30] registered [ 0.357122] acpiphp: Slot [31] registered [ 0.358000] PCI host bridge to bus 0000:00 [ 0.358000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.360021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.363041] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.365020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.367019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.370026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.372168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.375062] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.378371] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.388018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.394942] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.397016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.399014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.401016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.403556] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.406977] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.409040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.411503] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.417017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.429000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.434000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.441386] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.449021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.457023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.477037] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.488469] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.501016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.507015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.520016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.531446] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.539022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.545021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.565020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.575697] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.581016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.586000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.607014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.615562] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.622016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.631017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.652071] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.662830] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.672015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.681017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.701016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.715075] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.717356] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.718800] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.725459] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.727239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.733127] iommu: Default domain type: Passthrough [ 0.735499] SCSI subsystem initialized [ 0.737148] ACPI: bus type USB registered [ 0.739106] usbcore: registered new interface driver usbfs [ 0.741084] usbcore: registered new interface driver hub [ 0.742103] usbcore: registered new device driver usb [ 0.744161] pps_core: LinuxPPS API ver. 1 registered [ 0.746011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.749060] PTP clock support registered [ 0.751143] EDAC MC: Ver: 3.0.0 [ 0.752925] PCI: Using ACPI for IRQ routing [ 0.755000] NetLabel: Initializing [ 0.757033] NetLabel: domain hash size = 128 [ 0.760015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.763113] NetLabel: unlabeled traffic allowed by default [ 0.765161] vgaarb: loaded [ 0.767300] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.769016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.777814] clocksource: Switched to clocksource kvm-clock [ 0.898174] VFS: Disk quotas dquot_6.6.0 [ 0.901259] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.903913] *** VALIDATE ramfs *** [ 0.906041] *** VALIDATE hugetlbfs *** [ 0.908419] pnp: PnP ACPI init [ 0.911863] pnp: PnP ACPI: found 6 devices [ 0.932280] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.936095] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.938895] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.941400] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.944339] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.947307] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.950826] NET: Registered protocol family 2 [ 0.953580] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.958968] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.963084] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.969671] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.974914] TCP: Hash tables configured (established 65536 bind 65536) [ 0.978598] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.983559] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.986985] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.990977] NET: Registered protocol family 1 [ 0.996926] RPC: Registered named UNIX socket transport module. [ 0.999204] RPC: Registered udp transport module. [ 1.001183] RPC: Registered tcp transport module. [ 1.003108] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.005546] NET: Registered protocol family 44 [ 1.007408] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.010796] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.013568] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.017250] PCI: CLS 0 bytes, default 64 [ 1.021826] Unpacking initramfs... [ 2.483198] debug: unmapping init [mem 0xffff9b42fcc54000-0xffff9b42fffbffff] [ 2.488514] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.492013] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.496599] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.993188] Initialise system trusted keyrings [ 2.995176] Key type blacklist registered [ 2.997170] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.008159] zbud: loaded [ 3.011486] *** VALIDATE nfs *** [ 3.012753] *** VALIDATE nfs4 *** [ 3.014358] pstore: using deflate compression [ 3.018650] Platform Keyring initialized [ 3.143781] NET: Registered protocol family 38 [ 3.146310] Key type asymmetric registered [ 3.149018] Asymmetric key parser 'x509' registered [ 3.151478] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.154656] io scheduler mq-deadline registered [ 3.156419] io scheduler kyber registered [ 3.158387] io scheduler bfq registered [ 3.160478] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.163773] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.166958] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.170617] ACPI: Power Button [PWRF] [ 3.178122] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.185844] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.198953] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.207054] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.220830] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.251606] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.279363] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.284248] Non-volatile memory driver v1.3 [ 3.286201] Linux agpgart interface v0.103 [ 3.318779] virtio_blk virtio1: [vda] 142768 512-byte logical blocks (73.1 MB/69.7 MiB) [ 3.323806] vda: detected capacity change from 0 to 73097216 [ 3.352230] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.355657] vdb: detected capacity change from 0 to 1073741824 [ 3.376369] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.380367] vdc: detected capacity change from 0 to 2621440000 [ 3.400155] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.403205] vdd: detected capacity change from 0 to 2621440000 [ 3.422506] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.426114] vde: detected capacity change from 0 to 4294967296 [ 3.444296] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.447619] vdf: detected capacity change from 0 to 4294967296 [ 3.458665] libphy: Fixed MDIO Bus: probed [ 3.472387] usbcore: registered new interface driver usbserial_generic [ 3.475652] usbserial: USB Serial support registered for generic [ 3.478299] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.483523] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.485616] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.488513] mousedev: PS/2 mouse device common for all mice [ 3.494411] rtc_cmos 00:05: RTC can wake from S4 [ 3.497283] rtc_cmos 00:05: registered as rtc0 [ 3.498986] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.499496] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.499534] intel_pstate: CPU model not supported [ 3.501778] hid: raw HID events driver (C) Jiri Kosina [ 3.513832] usbcore: registered new interface driver usbhid [ 3.515747] usbhid: USB HID core driver [ 3.517660] drop_monitor: Initializing network drop monitor service [ 3.520733] Initializing XFRM netlink socket [ 3.522581] NET: Registered protocol family 10 [ 3.523183] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.525972] Segment Routing with IPv6 [ 3.534166] NET: Registered protocol family 17 [ 3.536660] mpls_gso: MPLS GSO support [ 3.538143] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.542970] RAS: Correctable Errors collector initialized. [ 3.545263] AVX version of gcm_enc/dec engaged. [ 3.546705] AES CTR mode by8 optimization enabled [ 3.626794] sched_clock: Marking stable (3626772086, 0)->(4536215227, -909443141) [ 3.631251] registered taskstats version 1 [ 3.633533] Loading compiled-in X.509 certificates [ 3.635849] zswap: loaded using pool lzo/zbud [ 3.663031] Key type big_key registered [ 3.675617] Key type encrypted registered [ 3.677526] ima: No TPM chip found, activating TPM-bypass! [ 3.680225] ima: Allocated hash algorithm: sha1 [ 3.682714] ima: No architecture policies found [ 3.684573] evm: Initialising EVM extended attributes: [ 3.686898] evm: security.selinux [ 3.688510] evm: security.ima [ 3.689784] evm: security.capability [ 3.691457] evm: HMAC attrs: 0x1 [ 3.694539] rtc_cmos 00:05: setting system clock to 2026-09-11 07:03:52 UTC (1789110232) [ 3.702294] debug: unmapping init [mem 0xffffffffb6003000-0xffffffffb61fffff] [ 3.705454] debug: unmapping init [mem 0xffffffffb4d82000-0xffffffffb5058fff] [ 3.712243] Write protecting the kernel read-only data: 28672k [ 3.715800] debug: unmapping init [mem 0xffffffffb3403000-0xffffffffb35fffff] [ 3.719171] debug: unmapping init [mem 0xffffffffb3d14000-0xffffffffb3dfffff] [ 3.758992] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.767080] systemd[1]: Detected virtualization kvm. [ 3.769096] systemd[1]: Detected architecture x86-64. [ 3.770899] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.799752] systemd[1]: No hostname configured. [ 3.801070] systemd[1]: Set hostname to . [ 3.802907] random: systemd: uninitialized urandom read (16 bytes read) [ 3.805065] systemd[1]: Initializing machine ID from random generator. [ 3.861988] random: ln: uninitialized urandom read (6 bytes read) [ 3.940647] random: systemd: uninitialized urandom read (16 bytes read) [ 3.944190] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.950337] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.953711] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.676825] device-mapper: uevent: version 1.0.3 [ 4.679499] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.408489] virtio_net virtio0 ens2: renamed from eth0 [ 5.437497] random: fast init done [ 5.447989] scsi host0: ata_piix [ 5.489239] scsi host1: ata_piix [ 5.491021] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.493530] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.159932] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.197233] random: crng init done [ 10.200022] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.702902] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.983033] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.264133] SELinux: Disabled at runtime. [ 12.329516] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.339891] systemd[1]: Detected virtualization kvm. [ 12.342061] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.883865] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.887178] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.893084] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.897405] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.900708] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.907311] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.911541] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on[ 13.040649] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ 13.354855] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.741142] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.820051] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.960701] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.971361] EDAC sbridge: Ver: 1.1.2 [ 15.755854] Key type dns_resolver registered [ 16.078539] NFS: Registering the id_resolver key type [ 16.081554] Key type id_resolver registered [ 16.083037] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg444-server login: [ 29.275069] hrtimer: interrupt took 4298949 ns [ 74.806414] libcfs: loading out-of-tree module taints kernel. [ 74.828574] Key type ._llcrypt registered [ 74.830342] Key type .llcrypt registered [ 74.931728] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 97.025592] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 98.316109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.327568] alg: No test for adler32 (adler32-zlib) [ 99.885883] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 100.513481] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 102.287243] Key type lgssc registered [ 104.801354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 111.582820] alg: No test for zstdfast (zstdfast-lustre-generic) [ 111.594700] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 128.487916] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 176.228877] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 191.669126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 191.707553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.007184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.036429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.097441] Lustre: lustre-MDT0000: new disk, initializing [ 193.165134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.181249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.716182] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.899381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.957962] Lustre: 6591:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.983395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 210.986899] Lustre: Skipped 1 previous similar message [ 211.088335] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 211.134750] Lustre: lustre-MDT0001: new disk, initializing [ 211.250689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.281358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 211.293223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.354280] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.478726] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 238.030229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.608131] Lustre: lustre-OST0000: new disk, initializing [ 238.619218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 238.628301] Lustre: 8494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.912830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.303135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.310760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 246.369200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 246.758147] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.744078] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.886201] Lustre: lustre-OST0001: new disk, initializing [ 259.889443] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 259.893906] Lustre: 9551:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.936567] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 265.598590] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.353836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 269.369161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 269.497115] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 278.584808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 287.458989] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.719833] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing check_logdir /tmp/testlogs/ [ 300.885379] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing yml_node [ 306.935828] Lustre: DEBUG MARKER: Client: 2.17.53.26 [ 311.043378] Lustre: DEBUG MARKER: MDS: 2.17.53.26 [ 314.458572] Lustre: DEBUG MARKER: OSS: 2.17.53.26 [ 316.256337] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 03:09:03 EDT 2026 [ 339.398122] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 341.277461] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.907526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 359.913624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.931573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.442120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.443357] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.457236] Lustre: Skipped 2 previous similar messages [ 361.470054] Lustre: Skipped 2 previous similar messages [ 363.531101] Lustre: server umount lustre-MDT0000 complete [ 367.484117] LustreError: 9550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789110596 with bad export cookie 7183543146638624377 [ 367.492100] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 367.494849] LustreError: 9550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 371.683596] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.689814] LustreError: 6596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.689952] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.702209] Lustre: Skipped 1 previous similar message [ 371.729900] LustreError: 6596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 372.706651] LustreError: 6597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.735535] LustreError: 6597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 374.062791] Lustre: server umount lustre-MDT0001 complete [ 378.489993] Lustre: server umount lustre-OST0000 complete [ 383.247349] Lustre: server umount lustre-OST0001 complete [ 390.527762] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 401.676464] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 443.081707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.517385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 443.574896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 443.711513] Lustre: lustre-MDT0000: new disk, initializing [ 443.809144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.839291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 448.128524] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 459.033238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.148347] Lustre: 15083:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 459.194828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 459.198836] Lustre: Skipped 1 previous similar message [ 459.287555] Lustre: lustre-MDT0001: new disk, initializing [ 459.360407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 459.394085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 459.410268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 463.702974] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 475.319872] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 481.346786] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 490.218144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 490.593843] Lustre: lustre-OST0000: new disk, initializing [ 490.601121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 490.620632] Lustre: 16827:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 490.769025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.776913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 491.787078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 492.011860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 498.053917] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 507.957492] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 512.544820] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 512.869658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 517.907536] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 518.104907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 522.737537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.744114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.753666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.766806] Lustre: Skipped 1 previous similar message [ 522.784990] Lustre: Skipped 4 previous similar messages [ 526.137822] Lustre: server umount lustre-OST0000 complete [ 530.912657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 530.926817] LustreError: Skipped 1 previous similar message [ 530.930616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.946093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 544.223963] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 544.494839] Lustre: server umount lustre-MDT0000 complete [ 548.320655] LustreError: 15945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.345061] LustreError: 15945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 548.519952] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789110777 with bad export cookie 7183543146638625910 [ 548.522062] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.529969] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 548.910764] Lustre: server umount lustre-MDT0001 complete [ 553.375461] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:13:00 (1789110780) [ 556.046724] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 557.613897] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:13:04 (1789110784) [ 564.729055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.208184] LustreError: 19091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.266062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 569.530126] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 570.348263] LustreError: 19092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.624555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.031952] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 591.197289] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 594.747680] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 601.760291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 610.093300] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 616.674832] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 625.638893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 625.640932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.647154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 625.647162] Lustre: Skipped 6 previous similar messages [ 625.650817] LustreError: Skipped 1 previous similar message [ 625.657066] Lustre: Skipped 2 previous similar messages [ 631.462911] Lustre: server umount lustre-OST0000 complete [ 636.386495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 642.317756] Lustre: server umount lustre-MDT0000 complete [ 645.808964] LustreError: 19073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789110874 with bad export cookie 7183543146638626792 [ 645.820609] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 645.826552] LustreError: 19073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 646.157735] Lustre: server umount lustre-MDT0001 complete [ 656.421761] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 659.191384] Key type lgssc unregistered [ 659.513192] LNet: 22266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.522458] LNetError: 22266:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 659.540154] LNet: Removed LNI 192.168.204.144@tcp [ 660.664163] Key type .llcrypt unregistered [ 660.672535] Key type ._llcrypt unregistered [ 684.095405] Key type ._llcrypt registered [ 684.106150] Key type .llcrypt registered [ 684.236117] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 685.207962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 685.234131] alg: No test for adler32 (adler32-zlib) [ 686.303719] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 686.563532] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 688.223427] Key type lgssc registered [ 689.235395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 692.392147] alg: No test for zstdfast (zstdfast-lustre-generic) [ 692.421163] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 698.269508] loop: module loaded [ 698.376066] loop0: detected capacity change from 0 to 204800000 [ 698.616432] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 698.633888] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 698.899932] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 705.685611] loop0: detected capacity change from 0 to 204800000 [ 705.898349] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 705.914400] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 705.933887] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 711.183163] loop0: detected capacity change from 0 to 204800000 [ 717.485674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 717.515727] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 718.773323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 718.833454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 718.950940] Lustre: lustre-MDT0000: new disk, initializing [ 719.076152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.098662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 723.312589] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 731.778257] loop1: detected capacity change from 0 to 204800000 [ 738.290966] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 738.840715] Lustre: lustre-OST07c6: new disk, initializing [ 738.846715] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 738.849608] Lustre: Skipped 1 previous similar message [ 738.853568] Lustre: 25506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 738.923172] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 746.091807] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 747.271992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 747.279949] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 747.361690] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 752.230894] Lustre: 24511:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 752.246790] Lustre: 24511:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 752.264644] Lustre: 24511:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 752.272216] Lustre: 24511:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 752.289750] Lustre: 24511:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 752.297165] Lustre: 24511:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 768.512338] Lustre: 24509:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 768.527263] Lustre: 24509:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 768.542439] Lustre: 24509:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 768.560991] Lustre: 24509:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 768.570512] Lustre: 24509:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 768.578305] Lustre: 24509:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 774.241195] Lustre: 24510:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 774.253500] Lustre: 24510:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 774.267498] Lustre: 24510:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 774.278044] Lustre: 24510:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.285843] Lustre: 24510:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 774.294396] Lustre: 24510:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.304938] Lustre: 24510:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 774.313678] Lustre: 24510:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.324185] Lustre: 24510:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 774.330162] Lustre: 24510:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.343985] Lustre: 24510:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 774.352861] Lustre: 24510:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.592373] Lustre: 24509:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 777.611927] Lustre: 24509:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 777.626049] Lustre: 24509:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 777.636276] Lustre: 24509:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.662485] Lustre: 24509:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.678237] Lustre: 24509:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.697265] Lustre: 24509:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 777.727196] Lustre: 24509:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.738700] Lustre: 24509:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 777.753304] Lustre: 24509:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.764033] Lustre: 24509:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.780406] Lustre: 24509:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.291241] Lustre: 24509:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 786.301169] Lustre: 24509:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 786.314119] Lustre: 24509:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 786.321671] Lustre: 24509:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 786.335299] Lustre: 24509:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 786.344361] Lustre: 24509:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 786.356308] Lustre: 24509:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 786.365220] Lustre: 24509:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 786.372238] Lustre: 24509:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 786.381994] Lustre: 24509:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 786.394635] Lustre: 24509:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 786.405115] Lustre: 24509:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 795.950449] Lustre: 24510:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 795.958574] Lustre: 24510:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 795.963307] Lustre: 24510:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 795.967331] Lustre: 24510:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 795.976631] Lustre: 24510:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 795.982210] Lustre: 24510:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 795.989441] Lustre: 24510:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 795.996064] Lustre: 24510:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 796.002625] Lustre: 24510:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 796.011368] Lustre: 24510:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 796.015702] Lustre: 24510:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 796.020287] Lustre: 24510:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 844.768685] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 844.776496] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.801529] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 849.904126] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 855.019469] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 856.031172] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 856.219210] Lustre: server umount lustre-OST07c6 complete [ 866.794856] Lustre: server umount lustre-MDT0000 complete [ 887.145271] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 889.896586] Key type lgssc unregistered [ 890.267737] LNet: 28033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.276381] LNetError: 28033:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.301310] LNet: Removed LNI 192.168.204.144@tcp [ 891.316462] Key type .llcrypt unregistered [ 891.318912] Key type ._llcrypt unregistered [ 910.730290] Key type ._llcrypt registered [ 910.732892] Key type .llcrypt registered [ 910.821597] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 911.736879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 911.882060] alg: No test for adler32 (adler32-zlib) [ 912.918232] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 913.140033] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 914.855297] Key type lgssc registered [ 915.573841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 917.797871] alg: No test for zstdfast (zstdfast-lustre-generic) [ 917.809600] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 923.025289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 923.053521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.576350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.141228] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 934.882672] LustreError: 29373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.897079] LustreError: 29373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 936.862369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.190442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 941.012392] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 947.017375] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 951.260143] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 958.706100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 959.132185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.923630] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 969.404786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 972.698658] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 990.178975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 990.197119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 990.204020] Lustre: Skipped 1 previous similar message [ 994.781340] Lustre: server umount lustre-OST0000 complete [ 998.880812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 998.888755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.911017] Lustre: Skipped 1 previous similar message [ 998.920659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1000.416961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1000.418152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.055122] Lustre: server umount lustre-MDT0000 complete [ 1005.538286] LustreError: 29370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1005.565396] LustreError: 29370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1009.268355] LustreError: 29354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789111238 with bad export cookie 13123093783953874846 [ 1009.271359] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1009.275715] LustreError: 29354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1009.731940] Lustre: server umount lustre-MDT0001 complete [ 1022.938344] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1025.767956] Key type lgssc unregistered [ 1026.072706] LNet: 32753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.078496] LNetError: 32753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.101388] LNet: Removed LNI 192.168.204.144@tcp [ 1026.941149] Key type .llcrypt unregistered [ 1026.943805] Key type ._llcrypt unregistered [ 1046.191539] Key type ._llcrypt registered [ 1046.194724] Key type .llcrypt registered [ 1046.345544] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1047.240694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1047.275949] alg: No test for adler32 (adler32-zlib) [ 1048.257778] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 1048.553124] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1050.255177] Key type lgssc registered [ 1051.264788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1053.638802] alg: No test for zstdfast (zstdfast-lustre-generic) [ 1053.647270] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 1059.172323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1059.200721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.805719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.272847] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1071.075995] LustreError: 34092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1071.110928] LustreError: 34092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1074.052432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.469791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.208742] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1084.809746] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1088.593762] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1095.207175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1095.494549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1101.205034] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1105.777115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1106.377035] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1116.140591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.146245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1116.155903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1116.164621] Lustre: Skipped 1 previous similar message [ 1121.252409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.268892] Lustre: Skipped 1 previous similar message [ 1122.421433] Lustre: server umount lustre-OST0000 complete [ 1130.982890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1130.990041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.001962] Lustre: Skipped 1 previous similar message [ 1131.005326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1132.889743] Lustre: server umount lustre-MDT0000 complete [ 1136.470381] LustreError: 34071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789111365 with bad export cookie 15222326069872896798 [ 1136.484831] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1136.486695] LustreError: 34071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1136.973286] Lustre: server umount lustre-MDT0001 complete [ 1149.483427] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1152.349720] Key type lgssc unregistered [ 1152.768279] LNet: 37265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.776246] LNetError: 37265:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.785905] LNet: Removed LNI 192.168.204.144@tcp [ 1153.511803] Key type .llcrypt unregistered [ 1153.514156] Key type ._llcrypt unregistered [ 1179.475496] Key type ._llcrypt registered [ 1179.486735] Key type .llcrypt registered [ 1179.606179] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1180.491976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1180.563116] alg: No test for adler32 (adler32-zlib) [ 1181.601419] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 1181.883056] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1183.599188] Key type lgssc registered [ 1184.798925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1187.783094] alg: No test for zstdfast (zstdfast-lustre-generic) [ 1187.797751] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 1193.153560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1193.181126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.796462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.312638] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1205.218707] LustreError: 38749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.234361] LustreError: 38749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1206.884218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.176029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1210.817248] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1216.604498] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1220.032435] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1227.779341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1228.041033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.412105] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1240.369476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1240.665808] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1247.233732] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1248.224456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1248.243484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.265631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1250.788832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.791844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1250.804010] Lustre: Skipped 1 previous similar message [ 1252.703626] Lustre: server umount lustre-MDT0000 complete [ 1255.907691] LustreError: 38748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.935971] LustreError: 38748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1257.163992] LustreError: 39467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789111485 with bad export cookie 2935278928395321277 [ 1257.168706] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1257.176549] LustreError: 39467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1257.442056] LustreError: 38744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.464384] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 1257.473745] Lustre: Skipped 1 previous similar message [ 1261.052350] LustreError: 40958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.065244] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.091043] Lustre: Skipped 1 previous similar message [ 1262.048593] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1262.053778] Lustre: Skipped 1 previous similar message [ 1263.561709] Lustre: server umount lustre-MDT0001 complete [ 1284.634845] Lustre: server umount lustre-OST0000 complete [ 1300.748858] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1303.543428] Key type lgssc unregistered [ 1303.945807] LNet: 42254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.953503] LNetError: 42254:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.978891] LNet: Removed LNI 192.168.204.144@tcp [ 1304.943366] Key type .llcrypt unregistered [ 1304.945520] Key type ._llcrypt unregistered [ 1328.635869] Key type ._llcrypt registered [ 1328.638065] Key type .llcrypt registered [ 1328.755057] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1329.673619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1329.772540] alg: No test for adler32 (adler32-zlib) [ 1330.916925] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 1331.313620] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1332.967164] Key type lgssc registered [ 1334.066784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1336.508214] alg: No test for zstdfast (zstdfast-lustre-generic) [ 1336.533947] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 1341.181821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1341.199530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.605660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.558159] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1353.188947] LustreError: 43736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.205600] LustreError: 43736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1354.344030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.622025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1358.845070] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1364.234925] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1367.846857] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1374.011727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1374.320642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1380.673275] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1386.487424] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1387.625869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1397.242118] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1398.245236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.250498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1398.262088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1402.338781] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1402.346799] Lustre: Skipped 1 previous similar message [ 1402.691751] Lustre: server umount lustre-OST0000 complete [ 1421.282821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1421.302337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.313950] Lustre: Skipped 1 previous similar message [ 1421.319476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.846076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.386939] LustreError: 46351:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000e9a3ad39) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1427.567444] Lustre: server umount lustre-MDT0000 complete [ 1428.964857] LustreError: 45281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.993329] LustreError: 45281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1431.517956] LustreError: 43717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789111660 with bad export cookie 16819291780703015696 [ 1431.526540] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1431.532317] LustreError: 43717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1438.113300] Lustre: server umount lustre-MDT0001 complete [ 1451.112734] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1454.255516] Key type lgssc unregistered [ 1454.619751] LNet: 47087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.628748] LNetError: 47087:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.662634] LNet: Removed LNI 192.168.204.144@tcp [ 1455.408238] Key type .llcrypt unregistered [ 1455.417324] Key type ._llcrypt unregistered [ 1479.825971] Key type ._llcrypt registered [ 1479.829019] Key type .llcrypt registered [ 1479.922935] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1480.817464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1480.860035] alg: No test for adler32 (adler32-zlib) [ 1481.961825] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 1482.280907] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1484.047652] Key type lgssc registered [ 1485.267227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1488.202937] alg: No test for zstdfast (zstdfast-lustre-generic) [ 1488.214585] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 1493.028042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1493.055649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1494.509773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.826704] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1504.739652] LustreError: 48569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.758161] LustreError: 48569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1507.277130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.607366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1511.916238] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1517.140720] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1520.695241] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1527.060245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1527.439805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.594218] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1538.722775] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1539.772560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1544.699733] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1547.876023] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1560.598725] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-11 3h29m46s [ 1562.581493] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1565.670738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1565.674779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1565.689575] Lustre: Skipped 2 previous similar messages [ 1565.714480] Lustre: Skipped 2 previous similar messages [ 1566.756421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1568.284060] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1568.296431] Lustre: Skipped 3 previous similar messages [ 1571.563460] Lustre: server umount lustre-MDT0000 complete [ 1572.835710] LustreError: 49780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1572.850855] LustreError: 49780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1574.984091] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1578.988554] LustreError: 49281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.028030] LustreError: 49281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1581.270834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.427114] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1581.646236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.715802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1582.703255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1585.876545] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1589.472570] Lustre: DEBUG MARKER: Wait for df (36654) ... [ 1590.991630] Lustre: DEBUG MARKER: done [ 1596.899964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1596.908157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.919733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1598.436242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.338358] Lustre: server umount lustre-OST0000 complete [ 1608.677528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.686772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.692314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1608.693537] Lustre: Skipped 1 previous similar message [ 1608.740406] Lustre: Skipped 3 previous similar messages [ 1611.259718] Lustre: server umount lustre-MDT0000 complete [ 1615.220471] LustreError: 48550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789111844 with bad export cookie 2011756285874061997 [ 1615.226977] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1615.487180] Lustre: server umount lustre-MDT0001 complete [ 1627.029436] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1629.822959] Key type lgssc unregistered [ 1630.102382] LNet: 53118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.113350] LNetError: 53118:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1630.133751] LNet: Removed LNI 192.168.204.144@tcp [ 1630.769815] Key type .llcrypt unregistered [ 1630.773067] Key type ._llcrypt unregistered [ 1668.352320] Key type ._llcrypt registered [ 1668.355840] Key type .llcrypt registered [ 1668.555751] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1669.835558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1669.986238] alg: No test for adler32 (adler32-zlib) [ 1671.098878] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 1671.308326] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1672.975193] Key type lgssc registered [ 1674.174106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1676.810504] alg: No test for zstdfast (zstdfast-lustre-generic) [ 1676.822158] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 1681.313913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1681.336080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1682.701703] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1682.721993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1682.882360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.278199] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1693.797671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1693.895828] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1693.931220] Lustre: 55391:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1694.157065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1698.409533] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1703.417944] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1706.945919] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1713.576127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1713.754849] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1713.952198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1719.559661] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1720.468305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1724.900910] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1731.921939] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-11 3h32m39s [ 1733.570458] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1741.086967] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1743.336647] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1743.531713] LustreError: 55398:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1763.856905] Lustre: lustre-MDT0000: Client 7054bbc9-5ce2-47c6-8a8c-59d4351e4505 (at 192.168.204.44@tcp) reconnecting [ 1783.543275] LustreError: 55398:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1785.584213] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1787.848529] Lustre: DEBUG MARKER: done [ 1796.411884] Lustre: server umount lustre-OST0000 complete [ 1801.696908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1801.704801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.735154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.786409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.814923] Lustre: Skipped 2 previous similar messages [ 1808.887914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1808.904502] Lustre: Skipped 1 previous similar message [ 1812.110583] Lustre: server umount lustre-MDT0000 complete [ 1812.963888] Lustre: 54662:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789112025/real 1789112025] req@00000000039ace41 x1876019799709568/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789112041 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1813.029916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1813.051897] Lustre: Skipped 1 previous similar message [ 1816.312886] LustreError: 55384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112045 with bad export cookie 1641650510503274088 [ 1816.317036] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1816.329215] LustreError: 55384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1816.568236] Lustre: server umount lustre-MDT0001 complete [ 1826.310528] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1828.652731] Key type lgssc unregistered [ 1828.916955] LNet: 59365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.925289] LNetError: 59365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.951795] LNet: Removed LNI 192.168.204.144@tcp [ 1829.713848] Key type .llcrypt unregistered [ 1829.720260] Key type ._llcrypt unregistered [ 1877.484720] Key type ._llcrypt registered [ 1877.487785] Key type .llcrypt registered [ 1877.584968] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1878.692417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1878.728084] alg: No test for adler32 (adler32-zlib) [ 1879.818666] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 1880.096625] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1881.824064] Key type lgssc registered [ 1883.142979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1885.831830] alg: No test for zstdfast (zstdfast-lustre-generic) [ 1885.838312] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 1886.574355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1886.587591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.811395] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1887.830484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1887.970573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.077058] Lustre: Failing over lustre-MDT0000 [ 1889.320714] Lustre: server umount lustre-MDT0000 complete [ 1895.745417] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:35:22 (1789112122) [ 1902.642810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.192545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.133967] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1917.845415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.982927] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1918.030393] Lustre: 62231:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1918.362159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1922.754437] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1927.814189] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1931.641441] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1939.384662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1939.584567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1939.762741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.514236] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1948.375101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1951.355550] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1956.120928] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1964.007567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.017580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.022834] Lustre: Skipped 1 previous similar message [ 1968.699288] Lustre: server umount lustre-MDT0000 complete [ 1969.123964] LustreError: 62245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.132068] LustreError: 62245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1972.461746] LustreError: 62225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112201 with bad export cookie 5316782081050545960 [ 1972.464913] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1972.472093] LustreError: 62225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1972.704662] Lustre: server umount lustre-MDT0001 complete [ 1976.346256] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1981.110172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1989.676140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.559681] Lustre: 61107:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789112203/real 1789112203] req@000000009087cece x1876020017797248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789112219 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1990.582864] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.596541] Lustre: Skipped 1 previous similar message [ 1993.695541] Lustre: 61104:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789112206/real 1789112206] req@000000001f0004f3 x1876020017797376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789112222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1995.743355] Lustre: 61105:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789112208/real 1789112208] req@000000006094829b x1876020017797504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789112224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1997.793313] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49c90122192e180b [ 1997.798036] Lustre: MGC192.168.204.144@tcp: Connection restored to 0@lo (at 0@lo) [ 1998.112706] LustreError: 63856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.209666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.300935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1999.199305] Lustre: 61107:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789112211/real 1789112211] req@0000000063264eeb x1876020017797760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789112227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1999.201168] LustreError: 63857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.801973] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2003.297335] Lustre: 61106:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789112216/real 1789112216] req@00000000316d2ffb x1876020017797888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789112232 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2005.474797] LustreError: 63857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.490919] LustreError: 63857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2008.897629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.363332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2014.489486] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2014.705124] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2019.825811] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2020.102811] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2024.355431] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2035.170327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.187489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2035.188513] Lustre: Skipped 2 previous similar messages [ 2035.189774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2035.200350] Lustre: Skipped 4 previous similar messages [ 2036.590684] Lustre: server umount lustre-MDT0000 complete [ 2040.018218] LustreError: 62952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112268 with bad export cookie 5316782081050548235 [ 2040.022865] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2040.027856] LustreError: 62952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2040.295891] Lustre: server umount lustre-MDT0001 complete [ 2046.297510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2056.608439] Lustre: 61105:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789112269/real 1789112269] req@00000000420da894 x1876020017822464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789112285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2056.641144] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.786516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.873428] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49c90122192e1d6e [ 2064.912213] Lustre: MGC192.168.204.144@tcp: Connection restored to 0@lo (at 0@lo) [ 2065.261911] LustreError: 66380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.347984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.426110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2069.583884] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2075.631780] LustreError: 68684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.636363] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2075.649497] LustreError: 68684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2078.194881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.072198] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2083.829254] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2088.824522] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2093.283839] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2104.296509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.300451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2104.301921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.309373] Lustre: Skipped 2 previous similar messages [ 2106.304283] Lustre: server umount lustre-MDT0000 complete [ 2109.411386] LustreError: 66380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.431261] LustreError: 66380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2109.673400] LustreError: 62226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112338 with bad export cookie 5316782081050549614 [ 2109.676233] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.681269] LustreError: 62226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2109.967933] Lustre: server umount lustre-MDT0001 complete [ 2116.742170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2120.851963] Lustre: DEBUG MARKER: files compared the same [ 2133.304148] Lustre: server umount lustre-OST0000 complete [ 2152.417877] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2156.060975] Key type lgssc unregistered [ 2156.494409] LNet: 71151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.512300] LNetError: 71151:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2156.548643] LNet: Removed LNI 192.168.204.144@tcp [ 2157.924713] Key type .llcrypt unregistered [ 2157.926334] Key type ._llcrypt unregistered [ 2182.168655] Key type ._llcrypt registered [ 2182.170984] Key type .llcrypt registered [ 2182.237929] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2182.883208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.955192] alg: No test for adler32 (adler32-zlib) [ 2184.091239] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 2184.331978] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2185.983190] Key type lgssc registered [ 2187.025334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2189.161569] alg: No test for zstdfast (zstdfast-lustre-generic) [ 2189.187912] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 2194.034805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2194.063167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.665309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.513241] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 2206.182232] LustreError: 72635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.195043] LustreError: 72635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2207.651557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.845459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2211.788289] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 2218.268644] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2222.930177] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2230.168813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2230.436257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.130162] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 2240.682189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2241.517739] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2251.234125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2251.237846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.249199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2251.274279] Lustre: Skipped 1 previous similar message [ 2256.359815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2256.365663] Lustre: Skipped 2 previous similar messages [ 2256.933612] Lustre: server umount lustre-OST0000 complete [ 2264.035257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2264.042763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.056865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2266.593732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.599765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2266.628419] Lustre: Skipped 1 previous similar message [ 2268.661516] Lustre: server umount lustre-MDT0000 complete [ 2271.712308] LustreError: 72634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.722016] LustreError: 72634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2272.266334] LustreError: 72614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112501 with bad export cookie 5101633791013328991 [ 2272.275955] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2272.280903] LustreError: 72614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2272.582122] Lustre: server umount lustre-MDT0001 complete [ 2284.533552] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2287.507726] Key type lgssc unregistered [ 2287.790816] LNet: 75810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2287.803503] LNetError: 75810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.818835] LNet: Removed LNI 192.168.204.144@tcp [ 2288.730444] Key type .llcrypt unregistered [ 2288.732644] Key type ._llcrypt unregistered [ 2315.842088] Key type ._llcrypt registered [ 2315.844386] Key type .llcrypt registered [ 2315.990274] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2317.803184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2317.855108] alg: No test for adler32 (adler32-zlib) [ 2318.929913] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 2319.271116] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2320.967187] Key type lgssc registered [ 2322.366888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2325.963561] alg: No test for zstdfast (zstdfast-lustre-generic) [ 2325.981480] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 2331.710676] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2331.751876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2347.359438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2352.479280] LustreError: 77268:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 2379.167153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.546883] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2390.056683] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2397.392879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.590743] Lustre: *** cfs_fail_loc=706, val=0*** [ 2398.221978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.339994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2403.306997] LustreError: 77294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.330259] LustreError: 77294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2403.371209] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2408.427491] LustreError: 77293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.539299] LustreError: 77294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.568626] LustreError: 77294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2414.898523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.416645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2419.667807] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2426.040350] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2431.144763] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2434.018261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2434.029246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.054822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2436.065869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2436.067257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2436.068638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.186421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2441.190502] Lustre: Skipped 2 previous similar messages [ 2446.309545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2446.316459] Lustre: Skipped 1 previous similar message [ 2447.327504] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2447.434774] Lustre: server umount lustre-OST0000 complete [ 2451.425299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.431543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2451.445075] Lustre: Skipped 1 previous similar message [ 2457.326383] Lustre: server umount lustre-MDT0000 complete [ 2461.514523] LustreError: 77276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112690 with bad export cookie 7308231292238634911 [ 2461.515218] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2461.534422] LustreError: 77276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2461.788291] Lustre: server umount lustre-MDT0001 complete [ 2474.276509] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2476.859570] Key type lgssc unregistered [ 2477.119704] LNet: 80530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2477.130186] LNetError: 80530:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2478.182456] LNet: Removed LNI 192.168.204.144@tcp [ 2478.788104] Key type .llcrypt unregistered [ 2478.793456] Key type ._llcrypt unregistered [ 2504.302832] Key type ._llcrypt registered [ 2504.304772] Key type .llcrypt registered [ 2504.400218] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2505.365585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2505.518372] alg: No test for adler32 (adler32-zlib) [ 2506.714658] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 2507.019737] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2508.703189] Key type lgssc registered [ 2509.805940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2512.672398] alg: No test for zstdfast (zstdfast-lustre-generic) [ 2512.693988] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 2517.822290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2517.838860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.689833] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2531.715600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.171530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2537.116461] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2537.464570] LustreError: 82683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.491126] LustreError: 82683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2541.543142] LustreError: 82687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2541.567668] LustreError: 82687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2546.663574] LustreError: 82686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.687078] LustreError: 82686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2546.964831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2547.471975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.801893] LustreError: 82687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.825910] LustreError: 82687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2554.292709] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2556.902624] LustreError: 82686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.922373] LustreError: 82686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2563.986970] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2564.014452] Lustre: MGS: Client 01c50dd5-8a99-4ded-87f9-f1441e4e5a9c (at 0@lo) reconnecting [ 2564.038791] LustreError: 82697:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2564.267416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2564.367395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2568.570433] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2581.472587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2581.477321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.492070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2585.056219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2585.058886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2585.064144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.085413] Lustre: Skipped 1 previous similar message [ 2587.421562] Lustre: server umount lustre-OST0000 complete [ 2595.296922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.302282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2595.308115] Lustre: Skipped 1 previous similar message [ 2595.311081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2595.315433] Lustre: Skipped 1 previous similar message [ 2597.100885] Lustre: server umount lustre-MDT0000 complete [ 2600.418581] LustreError: 82687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.436785] LustreError: 82687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2601.093444] Lustre: server umount lustre-MDT0001 complete [ 2605.076136] Lustre: server umount MGS complete [ 2617.223136] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:47:24 (1789112844) [ 2648.634439] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2658.026159] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2696.330739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.295649] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2711.819275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.049068] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2712.106102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2712.198748] Lustre: lustre-MDT0001: new disk, initializing [ 2712.284888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2712.309737] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2712.320311] Lustre: 89765:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2712.328872] LustreError: 89765:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2717.134945] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2729.856536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2730.215272] Lustre: lustre-OST0000: new disk, initializing [ 2730.221347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2730.230486] Lustre: 90515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2730.338986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2736.948881] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2740.330525] Lustre: 90954:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000ca17ee40 x1876020675256704/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789112985 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2745.585900] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2753.592478] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2753.605295] Lustre: MGS: Client 0fd2d28a-4a6f-4a3f-b733-fbaf33b039da (at 0@lo) reconnecting [ 2753.619978] LustreError: 89751:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2753.644847] Lustre: 89048:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2753.693914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2753.786826] Lustre: lustre-MDT0000: new disk, initializing [ 2753.981420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.024891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2754.159491] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2755.104138] Lustre: 91471:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000a9153d00 x1876020675263360/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789112999 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2758.879500] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2760.676856] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2764.209307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2764.222887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2764.363738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2790.371378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2790.383805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.401377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2795.407973] Lustre: server umount lustre-OST0000 complete [ 2804.192346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2804.196819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.199423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.199432] Lustre: Skipped 2 previous similar messages [ 2804.203013] LustreError: Skipped 1 previous similar message [ 2804.242656] Lustre: Skipped 2 previous similar messages [ 2806.397986] Lustre: server umount lustre-MDT0000 complete [ 2809.315758] LustreError: 89737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.328897] LustreError: 89737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2816.718800] Lustre: server umount lustre-MDT0001 complete [ 2820.884592] Lustre: server umount MGS complete [ 2836.004120] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:51:02 (1789113062) [ 2856.896362] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2859.979847] Key type lgssc unregistered [ 2860.347048] LNet: 93940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2860.359521] LNetError: 93940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2860.385583] LNet: Removed LNI 192.168.204.144@tcp [ 2861.377828] Key type .llcrypt unregistered [ 2861.381700] Key type ._llcrypt unregistered [ 2879.603717] Key type ._llcrypt registered [ 2879.606247] Key type .llcrypt registered [ 2879.691890] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2880.639693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2880.664104] alg: No test for adler32 (adler32-zlib) [ 2881.642612] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 2881.822560] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2883.431128] Key type lgssc registered [ 2884.550374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2886.985653] alg: No test for zstdfast (zstdfast-lustre-generic) [ 2886.997107] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 2891.948417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2891.972993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.471599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2898.309509] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2904.042303] LustreError: 95231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.059555] LustreError: 95231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2908.166250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.673852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2913.896424] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2921.313817] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2925.498078] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2932.676687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2933.150645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2938.295396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2939.654724] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2945.376514] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2954.721073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2954.734889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.765882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2958.822485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2958.825937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2958.842819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.946449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2963.952805] Lustre: Skipped 2 previous similar messages [ 2968.033608] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2968.231519] Lustre: server umount lustre-OST0000 complete [ 2974.180867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.192505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.200571] Lustre: Skipped 1 previous similar message [ 2978.770286] Lustre: server umount lustre-MDT0000 complete [ 2979.301729] LustreError: 95944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.320238] LustreError: 95944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2982.416709] LustreError: 95210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789113211 with bad export cookie 7954914173584371096 [ 2982.419279] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2982.427299] LustreError: 95210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2982.771527] Lustre: server umount lustre-MDT0001 complete [ 2994.246521] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2997.228336] Key type lgssc unregistered [ 2997.554353] LNet: 98408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2997.572907] LNetError: 98408:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2998.636813] LNet: Removed LNI 192.168.204.144@tcp [ 2999.499772] Key type .llcrypt unregistered [ 2999.502604] Key type ._llcrypt unregistered [ 3019.178759] Key type ._llcrypt registered [ 3019.181373] Key type .llcrypt registered [ 3019.309466] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3020.444594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3020.475144] alg: No test for adler32 (adler32-zlib) [ 3021.607863] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 3021.881396] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3023.530096] Key type lgssc registered [ 3024.970640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3027.314830] alg: No test for zstdfast (zstdfast-lustre-generic) [ 3027.325842] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 3031.469600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3031.484892] LustreError: 99559:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3031.503370] LustreError: 99560:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3031.522373] LustreError: 99559:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3031.529486] LustreError: 99559:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3031.543648] LustreError: 99559:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3031.560923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3033.028796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3040.925992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.454578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3047.033434] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3053.425365] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing lsmod [ 3056.709236] LustreError: 100731:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3056.717880] LustreError: 100732:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3056.724025] LustreError: 100731:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 3056.733884] LustreError: 100731:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3056.740629] LustreError: 100731:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3056.755470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3057.199971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.231083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3062.246279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3062.262231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.286655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3063.283370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.297714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3063.320373] Lustre: Skipped 1 previous similar message [ 3063.343101] Lustre: Skipped 2 previous similar messages [ 3066.402174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3066.406079] Lustre: Skipped 2 previous similar messages [ 3066.720677] Lustre: server umount lustre-MDT0000 complete [ 3071.177699] LustreError: 99574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789113299 with bad export cookie 3082513513995347025 [ 3071.183264] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3071.198547] LustreError: 99574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3071.459965] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3071.461459] LustreError: 99592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.476228] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3071.493081] LustreError: 99592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3072.543392] LustreError: 100028:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000027f732bf x1876021215254016/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3072.869887] Lustre: server umount lustre-MDT0001 complete [ 3077.155497] Lustre: server umount lustre-OST0000 complete [ 3085.680080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.150898] LustreError: 101795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.219661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.298419] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3091.425554] LustreError: 101796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.524211] LustreError: 101795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.854739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.395331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3104.524287] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3111.750774] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3115.735477] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3123.180036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3123.536645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3129.830075] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3130.735750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3135.723456] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3145.184274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3145.191385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.206601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3146.226361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3148.742533] Lustre: server umount lustre-OST0000 complete [ 3155.936417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3155.940126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.946397] Lustre: Skipped 1 previous similar message [ 3155.950036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3155.954078] Lustre: Skipped 2 previous similar messages [ 3158.789297] Lustre: server umount lustre-MDT0000 complete [ 3161.569524] LustreError: 101790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.586779] LustreError: 101790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3161.880811] LustreError: 101776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789113390 with bad export cookie 3082513513995347956 [ 3161.885859] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3161.893408] LustreError: 101776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3162.144664] Lustre: server umount lustre-MDT0001 complete [ 3174.829676] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3177.917698] Key type lgssc unregistered [ 3178.254628] LNet: 104973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3178.261548] LNetError: 104973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3178.276762] LNet: Removed LNI 192.168.204.144@tcp [ 3179.210991] Key type .llcrypt unregistered [ 3179.217112] Key type ._llcrypt unregistered [ 3208.625632] Key type ._llcrypt registered [ 3208.628849] Key type .llcrypt registered [ 3208.740819] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3209.978358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3210.047262] alg: No test for adler32 (adler32-zlib) [ 3211.216780] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 3211.446936] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3213.135613] Key type lgssc registered [ 3214.389883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3217.325801] alg: No test for zstdfast (zstdfast-lustre-generic) [ 3217.341835] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 3222.667967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3222.693603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.308873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.010439] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3234.785698] LustreError: 106457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.809337] LustreError: 106457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3238.659861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.143296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3244.103780] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3250.030394] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3254.895629] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3263.876019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.201531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3269.402283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3272.019110] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3279.033515] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3287.362325] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3291.707893] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3299.300518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3299.303599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.313649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3299.329183] Lustre: Skipped 1 previous similar message [ 3302.995631] Lustre: server umount lustre-OST0000 complete [ 3309.542464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.550592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.561164] Lustre: Skipped 1 previous similar message [ 3309.579092] Lustre: Skipped 2 previous similar messages [ 3313.191373] Lustre: server umount lustre-MDT0000 complete [ 3314.658536] LustreError: 106458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.677315] LustreError: 106458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3317.144950] LustreError: 106437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789113545 with bad export cookie 4541367384313132798 [ 3317.154424] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3317.158917] LustreError: 106437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3317.533251] Lustre: server umount lustre-MDT0001 complete [ 3329.085890] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3331.947243] Key type lgssc unregistered [ 3332.238401] LNet: 109739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3332.245746] LNetError: 109739:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3332.256552] LNet: Removed LNI 192.168.204.144@tcp [ 3333.159053] Key type .llcrypt unregistered [ 3333.161460] Key type ._llcrypt unregistered [ 3353.029122] Key type ._llcrypt registered [ 3353.031828] Key type .llcrypt registered [ 3353.192499] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3354.388427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3354.509856] alg: No test for adler32 (adler32-zlib) [ 3355.658541] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 3355.892301] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3357.559245] Key type lgssc registered [ 3358.932961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3361.651387] alg: No test for zstdfast (zstdfast-lustre-generic) [ 3361.663117] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 3367.426398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3367.456550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.062473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.322534] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3378.657393] LustreError: 111029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3378.687051] LustreError: 111029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3383.778231] LustreError: 111028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.092274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.460810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3391.029761] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3397.022991] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3400.811986] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3407.783576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3408.037411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3414.198526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3414.744571] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3420.540829] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3429.865937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3429.865968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.893263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3429.907981] Lustre: Skipped 1 previous similar message [ 3434.979234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3434.996752] Lustre: Skipped 1 previous similar message [ 3435.781878] Lustre: server umount lustre-OST0000 complete [ 3440.100993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.108122] Lustre: Skipped 1 previous similar message [ 3440.112933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3440.120950] Lustre: Skipped 1 previous similar message [ 3445.218251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3445.227842] Lustre: Skipped 1 previous similar message [ 3445.828144] Lustre: server umount lustre-MDT0000 complete [ 3449.165051] LustreError: 111008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789113677 with bad export cookie 13324925491068599581 [ 3449.173823] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3449.178801] LustreError: 111008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3449.414811] Lustre: server umount lustre-MDT0001 complete [ 3462.362798] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3465.299545] Key type lgssc unregistered [ 3465.622546] LNet: 114217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3465.628762] LNetError: 114217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3466.663273] LNet: Removed LNI 192.168.204.144@tcp [ 3467.685825] Key type .llcrypt unregistered [ 3467.690073] Key type ._llcrypt unregistered [ 3493.046842] Key type ._llcrypt registered [ 3493.051080] Key type .llcrypt registered [ 3493.150770] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3494.294645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3494.319886] alg: No test for adler32 (adler32-zlib) [ 3495.386628] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 3495.635595] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3497.295223] Key type lgssc registered [ 3498.554216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3501.411019] alg: No test for zstdfast (zstdfast-lustre-generic) [ 3501.445204] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 3506.697240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3506.738720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.383875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.233824] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3518.949489] LustreError: 115702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.963954] LustreError: 115702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3520.328561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.771890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3525.892063] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3533.046575] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3537.501171] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3545.161031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3545.580399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.185178] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3553.798499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3559.057481] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3567.137459] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3570.752508] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3570.755745] Lustre: Skipped 2 previous similar messages [ 3574.770934] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3574.777227] Lustre: Skipped 2 previous similar messages [ 3577.545966] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3577.592255] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3583.227333] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.44@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3583.237208] Lustre: Skipped 2 previous similar messages [ 3586.027124] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.44@tcp 192.168.0.[2,10]@tcp [ 3597.089883] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.44/24@tcp in log lustre-MDT0000 [ 3597.100642] Lustre: Skipped 2 previous similar messages [ 3602.500886] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.44/24@tcp [ 3602.508618] Lustre: Skipped 1 previous similar message [ 3613.161633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3613.183864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3613.227793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3615.266569] Lustre: server umount lustre-OST0000 complete [ 3620.320765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.322873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3620.349691] Lustre: Skipped 1 previous similar message [ 3620.361392] Lustre: Skipped 1 previous similar message [ 3625.440281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3625.447510] Lustre: Skipped 1 previous similar message [ 3629.730099] Lustre: server umount lustre-MDT0000 complete [ 3630.565079] LustreError: 115703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3630.586135] LustreError: 115703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3634.086089] LustreError: 116419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789113862 with bad export cookie 15068504453799555091 [ 3634.090548] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3634.105954] LustreError: 116419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3634.511674] Lustre: server umount lustre-MDT0001 complete [ 3646.874758] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3650.568023] Key type lgssc unregistered [ 3650.870251] LNet: 120184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3650.882970] LNetError: 120184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3650.899971] LNet: Removed LNI 192.168.204.144@tcp [ 3651.955316] Key type .llcrypt unregistered [ 3651.958793] Key type ._llcrypt unregistered [ 3685.019533] Key type ._llcrypt registered [ 3685.021984] Key type .llcrypt registered [ 3685.231458] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3686.734417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3686.911053] alg: No test for adler32 (adler32-zlib) [ 3688.169478] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 3688.480796] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3690.207185] Key type lgssc registered [ 3691.489534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3695.054319] alg: No test for zstdfast (zstdfast-lustre-generic) [ 3695.099558] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 3701.573536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3701.614790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.088256] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3703.115922] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.44/24@tcp [ 3703.141369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3707.718644] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3712.498214] LustreError: 121814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3712.542320] LustreError: 121814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3717.158469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.435914] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3717.452301] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.44/24@tcp [ 3717.495345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3722.255908] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3727.648592] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3731.328280] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3737.813962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3738.263161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3744.912324] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3747.545920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3751.192808] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3763.175333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.178400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3763.191681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.202664] LustreError: Skipped 1 previous similar message [ 3763.207754] Lustre: Skipped 1 previous similar message [ 3768.292628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3768.300780] Lustre: Skipped 1 previous similar message [ 3768.824970] Lustre: server umount lustre-OST0000 complete [ 3773.408832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3773.410909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3773.416442] Lustre: Skipped 2 previous similar messages [ 3773.427354] Lustre: Skipped 1 previous similar message [ 3778.529521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3778.544612] Lustre: Skipped 1 previous similar message [ 3778.831491] Lustre: server umount lustre-MDT0000 complete [ 3782.662994] LustreError: 121794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789114011 with bad export cookie 15492035905544988488 [ 3782.679099] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3783.112345] Lustre: server umount lustre-MDT0001 complete [ 3797.349628] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3800.518574] Key type lgssc unregistered [ 3801.005144] LNet: 125048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3801.012810] LNetError: 125048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3801.033079] LNet: Removed LNI 192.168.204.144@tcp [ 3802.063684] Key type .llcrypt unregistered [ 3802.070338] Key type ._llcrypt unregistered [ 3834.015326] Key type ._llcrypt registered [ 3834.019085] Key type .llcrypt registered [ 3834.139195] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 3850.010956] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3851.040166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3851.179952] alg: No test for adler32 (adler32-zlib) [ 3852.325970] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 3852.601938] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3854.239175] Key type lgssc registered [ 3855.357965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3858.181284] alg: No test for zstdfast (zstdfast-lustre-generic) [ 3858.197588] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 3896.165497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3896.202303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3897.657859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3897.720040] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3897.930317] Lustre: lustre-MDT0000: new disk, initializing [ 3898.106334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3898.126964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3903.607912] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3915.148348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.265286] Lustre: 128516:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3915.335884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3915.339970] Lustre: Skipped 1 previous similar message [ 3915.425707] Lustre: lustre-MDT0001: new disk, initializing [ 3915.591071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3915.623908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3915.634335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3920.118323] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3928.291728] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3933.358538] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3941.700145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3942.047879] Lustre: lustre-OST0000: new disk, initializing [ 3942.056522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3942.065828] Lustre: 130264:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3942.237063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.137887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3948.164996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3948.315769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3948.778639] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3956.962547] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3962.301699] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3962.491647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3967.165461] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3967.441997] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3974.112605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3974.112657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3974.135731] LustreError: Skipped 1 previous similar message [ 3974.143824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3974.156916] Lustre: Skipped 1 previous similar message [ 3975.970898] Lustre: server umount lustre-OST0000 complete [ 3982.306849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3982.317562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.329472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3982.338432] Lustre: Skipped 1 previous similar message [ 3984.355227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.358121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3984.384407] Lustre: Skipped 1 previous similar message [ 3986.546117] Lustre: server umount lustre-MDT0000 complete [ 3990.398052] LustreError: 128509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789114219 with bad export cookie 4768990807755872856 [ 3990.409414] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3990.417932] LustreError: 128509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3990.680695] Lustre: server umount lustre-MDT0001 complete [ 4000.880264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.551545] LustreError: 132336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4001.601418] LustreError: 132336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4001.715759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4006.728333] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4006.891877] LustreError: 132337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.000602] LustreError: 132336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.127648] LustreError: 132337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.256321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.820539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4023.142270] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4029.167538] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4033.194518] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4039.645915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4039.971315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4046.476395] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4052.692050] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4057.837752] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4058.126473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4063.274982] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4063.540464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4073.125274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4073.307484] Lustre: lustre-OST0001: new disk, initializing [ 4073.311391] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4073.317110] Lustre: 135335:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4073.437642] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4080.366403] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4081.801828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4081.816459] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4081.977856] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4089.706141] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4089.971729] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4095.271234] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4095.499739] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4100.366434] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99a8841d9000.ost_server_uuid 50 [ 4101.967525] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99a8841d9000.ost_server_uuid in IDLE state after 0 sec [ 4112.872943] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4112.879820] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.885238] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4112.908355] Lustre: Skipped 1 previous similar message [ 4112.931088] Lustre: Skipped 1 previous similar message [ 4113.882323] Lustre: server umount lustre-OST0001 complete [ 4117.985542] LustreError: 133934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4118.004477] LustreError: 133934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4119.009679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4119.019065] LustreError: Skipped 1 previous similar message [ 4119.024723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4119.048577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4124.341932] Lustre: server umount lustre-OST0000 complete [ 4128.231297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4128.234864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4128.244494] Lustre: Skipped 2 previous similar messages [ 4128.261341] Lustre: Skipped 4 previous similar messages [ 4134.382554] Lustre: server umount lustre-MDT0000 complete [ 4138.360658] LustreError: 135334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789114367 with bad export cookie 4768990807755873738 [ 4138.362813] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4138.375244] LustreError: 135334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4138.910701] Lustre: server umount lustre-MDT0001 complete [ 4160.789892] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4164.485315] Key type lgssc unregistered [ 4164.964162] LNet: 138179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4164.981458] LNetError: 138179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4165.013512] LNet: Removed LNI 192.168.204.144@tcp [ 4166.146120] Key type .llcrypt unregistered [ 4166.149341] Key type ._llcrypt unregistered [ 4209.827134] Key type ._llcrypt registered [ 4209.828484] Key type .llcrypt registered [ 4210.011141] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4226.925373] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4228.409222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4228.478320] alg: No test for adler32 (adler32-zlib) [ 4229.543308] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 4229.793884] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4231.479855] Key type lgssc registered [ 4232.833201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4236.262541] alg: No test for zstdfast (zstdfast-lustre-generic) [ 4236.295345] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 4271.733711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4271.802584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.186446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4273.240691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4273.386777] Lustre: lustre-MDT0000: new disk, initializing [ 4273.508096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.530825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4278.778959] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4289.423739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.610870] Lustre: 142390:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4289.681270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4289.684412] Lustre: Skipped 1 previous similar message [ 4289.750851] Lustre: lustre-MDT0001: new disk, initializing [ 4289.822529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4289.866571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4289.882718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4294.647834] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4302.644967] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4306.796640] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4314.331692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.608555] Lustre: lustre-OST0000: new disk, initializing [ 4314.613855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4314.630934] Lustre: 144133:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4314.756918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.832533] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4321.889863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4321.898901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4322.030874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4328.920382] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4334.352587] Lustre: Failing over lustre-OST0000 [ 4334.609961] Lustre: server umount lustre-OST0000 complete [ 4351.962798] Lustre: 140560:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789114564/real 1789114564] req@00000000d83b9fed x1876022481945344/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789114580 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4351.986332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4352.893597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4353.304416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.330419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4353.346622] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4353.511578] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4354.335148] Lustre: 140562:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789114566/real 1789114566] req@0000000089037e03 x1876022481946240/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789114582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4358.367527] Lustre: 140561:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789114571/real 1789114571] req@00000000e3e47795 x1876022481947392/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789114587 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4358.390096] Lustre: 140561:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4359.306544] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4361.482148] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4361.484791] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 4363.371285] Lustre: Failing over lustre-MDT0000 [ 4363.748477] Lustre: server umount lustre-MDT0000 complete [ 4364.383257] Lustre: 140560:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789114576/real 1789114576] req@00000000a7f92169 x1876022481948544/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789114592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4364.413497] Lustre: 140560:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4366.816122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4366.832480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.876423] LustreError: 142396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.913982] LustreError: 142396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4372.963761] LustreError: 142398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4372.984414] LustreError: 142398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4373.985456] LustreError: 142398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.083898] LustreError: 142401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.114786] LustreError: 142401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4382.298726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4382.375993] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4382.588684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.641647] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4385.246691] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4387.676708] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4387.824462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4387.830166] Lustre: Skipped 1 previous similar message [ 4387.857132] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4387.898680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4397.537898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4397.545719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.562210] Lustre: Skipped 2 previous similar messages [ 4397.567135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4398.071058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4398.077932] Lustre: Skipped 1 previous similar message [ 4402.903144] Lustre: server umount lustre-OST0000 complete [ 4408.291468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.293352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4408.294646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4408.304620] Lustre: Skipped 2 previous similar messages [ 4413.410048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4413.419659] Lustre: Skipped 2 previous similar messages [ 4413.965445] Lustre: server umount lustre-MDT0000 complete [ 4417.817562] LustreError: 142383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789114646 with bad export cookie 6947354420795413094 [ 4417.829554] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.831829] LustreError: 142383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4418.235143] Lustre: server umount lustre-MDT0001 complete [ 4429.987137] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4432.983936] Key type lgssc unregistered [ 4433.297586] LNet: 147621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4433.310483] LNetError: 147621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4433.329967] LNet: Removed LNI 192.168.204.144@tcp [ 4434.306756] Key type .llcrypt unregistered [ 4434.310548] Key type ._llcrypt unregistered [ 4474.782035] Key type ._llcrypt registered [ 4474.785420] Key type .llcrypt registered [ 4474.934808] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4476.406808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4476.476299] alg: No test for adler32 (adler32-zlib) [ 4477.667740] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 4477.924973] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4479.680417] Key type lgssc registered [ 4481.112235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4484.671577] alg: No test for zstdfast (zstdfast-lustre-generic) [ 4484.694688] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 4490.782833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4490.811234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.251142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4496.983947] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4502.503933] LustreError: 149209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4502.530290] LustreError: 149209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4505.785747] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4507.616179] LustreError: 149208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.947458] LustreError: 149237:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4509.962427] LustreError: 149237:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4509.971132] LustreError: 149237:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4510.349177] Lustre: server umount lustre-MDT0000 complete [ 4529.016819] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4539.446590] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4579.600831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4579.930394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4579.983558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4580.089097] Lustre: lustre-MDT0000: new disk, initializing [ 4580.186108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4580.223349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4585.239483] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4596.357617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.500437] Lustre: 152408:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4596.586546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4596.593823] Lustre: Skipped 1 previous similar message [ 4596.668547] Lustre: lustre-MDT0001: new disk, initializing [ 4596.822851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4596.881107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4596.888867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4601.223294] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4609.344754] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4613.586452] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4624.318308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4624.568691] Lustre: lustre-OST0000: new disk, initializing [ 4624.570879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4624.574175] Lustre: 154150:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4624.644378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.128045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4630.155585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4630.309749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4630.687270] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4641.161746] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4647.014921] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4649.520716] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:21:15 (1789114875) [ 4932.591405] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4932.591925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4932.614222] Lustre: Skipped 1 previous similar message [ 4937.697048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4937.710891] Lustre: Skipped 4 previous similar messages [ 4942.821749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4942.835541] Lustre: Skipped 2 previous similar messages [ 4946.399527] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4946.670600] Lustre: server umount lustre-MDT0000 complete [ 4947.941447] LustreError: 153190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.969338] LustreError: 153190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4950.820843] LustreError: 152399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789115179 with bad export cookie 15886008549712747685 [ 4950.822585] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4950.832574] LustreError: 152399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4953.056600] LustreError: 152415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4953.060363] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.095286] Lustre: Skipped 1 previous similar message [ 4953.104679] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4960.223829] LustreError: 152414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4960.223878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4960.235825] LustreError: 152414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4960.257612] Lustre: Skipped 1 previous similar message [ 4965.347431] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4965.811034] Lustre: server umount lustre-MDT0001 complete [ 4976.803900] Lustre: server umount lustre-OST0000 complete [ 4986.904893] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4997.318598] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5038.539050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5038.957268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5039.007328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5039.091978] Lustre: lustre-MDT0000: new disk, initializing [ 5039.218697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5039.236292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5043.503951] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5054.424995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.514707] Lustre: 158688:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5054.598466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5054.604481] Lustre: Skipped 1 previous similar message [ 5054.671132] Lustre: lustre-MDT0001: new disk, initializing [ 5054.732044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5054.754296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5054.765691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5059.042408] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5066.145109] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5070.434860] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5079.486307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5079.802210] Lustre: lustre-OST0000: new disk, initializing [ 5079.808561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5079.826110] Lustre: 160431:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5079.936075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5086.353551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5086.371866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5086.484959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5087.195619] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5095.917464] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5102.450617] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5102.760729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5107.464885] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5107.706722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5112.292070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5112.294083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5112.296566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5112.296574] Lustre: Skipped 1 previous similar message [ 5112.303290] LustreError: Skipped 1 previous similar message [ 5112.332389] Lustre: Skipped 1 previous similar message [ 5124.063257] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5124.322934] Lustre: server umount lustre-OST0000 complete [ 5131.747789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5131.756573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5131.770796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5131.778679] Lustre: Skipped 5 previous similar messages [ 5134.846534] Lustre: server umount lustre-MDT0000 complete [ 5137.890614] LustreError: 158696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5137.937178] LustreError: 158696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5139.248870] LustreError: 158682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789115368 with bad export cookie 15886008549712906515 [ 5139.255950] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5139.264114] LustreError: 158682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5139.715180] Lustre: server umount lustre-MDT0001 complete [ 5148.815145] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:29:35 (1789115375) [ 5165.156250] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5175.399944] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5219.368155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5219.625426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5219.653540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5219.748729] Lustre: lustre-MDT0000: new disk, initializing [ 5219.817898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5219.839163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5223.454619] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5236.643975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5236.765288] Lustre: 165020:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5237.072470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5241.948803] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5251.062240] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5256.076301] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5264.172567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5264.429023] Lustre: lustre-OST0000: new disk, initializing [ 5264.432628] Lustre: Skipped 1 previous similar message [ 5264.441660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5264.445741] Lustre: Skipped 2 previous similar messages [ 5264.450327] Lustre: 166771:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5270.660053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5270.676189] Lustre: Skipped 1 previous similar message [ 5270.686257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5270.741205] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5270.812369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5279.405160] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5291.500506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5291.501693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5291.517561] LustreError: Skipped 1 previous similar message [ 5291.519786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5291.530027] Lustre: Skipped 2 previous similar messages [ 5291.536216] Lustre: Skipped 2 previous similar messages [ 5293.963703] Lustre: server umount lustre-OST0000 complete [ 5298.656894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5301.732704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5301.742622] Lustre: Skipped 1 previous similar message [ 5304.280527] Lustre: server umount lustre-MDT0000 complete [ 5306.849378] LustreError: 166156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.879505] LustreError: 166156:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5308.606305] LustreError: 165013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789115537 with bad export cookie 15886008549712907404 [ 5308.609385] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5308.614814] LustreError: 165013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5309.144153] Lustre: server umount lustre-MDT0001 complete [ 5319.367981] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:32:26 (1789115546) [ 5337.060599] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5347.411612] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5384.545721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.780764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5384.814896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5384.890952] Lustre: lustre-MDT0000: new disk, initializing [ 5384.995696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.006986] Lustre: Skipped 2 previous similar messages [ 5385.036673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5389.397362] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5400.027156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5400.222963] Lustre: 170964:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5400.660153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5406.538371] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5414.720565] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5419.401820] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5426.931455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5427.184332] Lustre: 172714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5433.333465] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5434.435554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5434.571971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5440.513429] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5450.213340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.225432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.234307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5450.238583] Lustre: Skipped 1 previous similar message [ 5450.247640] Lustre: Skipped 4 previous similar messages [ 5455.117997] Lustre: server umount lustre-OST0000 complete [ 5465.996676] Lustre: server umount lustre-MDT0000 complete [ 5469.992262] LustreError: 170957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789115698 with bad export cookie 15886008549712908489 [ 5469.994105] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5470.007223] LustreError: 170957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5470.489561] Lustre: server umount lustre-MDT0001 complete [ 5482.011345] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5484.902870] Key type lgssc unregistered [ 5485.234702] LNet: 174439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5485.244714] LNetError: 174439:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5485.263078] LNet: Removed LNI 192.168.204.144@tcp [ 5485.922597] Key type .llcrypt unregistered [ 5485.924671] Key type ._llcrypt unregistered [ 5509.967343] Key type ._llcrypt registered [ 5509.969606] Key type .llcrypt registered [ 5510.078851] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5511.501953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5511.513994] alg: No test for adler32 (adler32-zlib) [ 5512.732248] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 5512.949571] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5514.616828] Key type lgssc registered [ 5516.232264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5519.242645] alg: No test for zstdfast (zstdfast-lustre-generic) [ 5519.257640] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 5525.663239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5525.691410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.420876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5532.132638] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5537.764140] LustreError: 175924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.784210] LustreError: 175924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5541.831381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.232309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5546.073804] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5552.263431] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5556.500311] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5564.527849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5564.847864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5571.378226] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5573.079941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5578.104334] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5597.676635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5597.691371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.716183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5598.693992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5598.694918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5598.695604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5603.158673] Lustre: server umount lustre-OST0000 complete [ 5608.930739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5608.937605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5608.942426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5608.952774] Lustre: Skipped 2 previous similar messages [ 5613.568622] Lustre: server umount lustre-MDT0000 complete [ 5614.049414] LustreError: 175925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.087792] LustreError: 175925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5617.544961] LustreError: 175906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789115846 with bad export cookie 8548580338034567816 [ 5617.549364] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5617.557121] LustreError: 175906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5617.965972] Lustre: server umount lustre-MDT0001 complete [ 5629.739574] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5632.830142] Key type lgssc unregistered [ 5633.145563] LNet: 179122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5633.158406] LNetError: 179122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5633.186653] LNet: Removed LNI 192.168.204.144@tcp [ 5634.049732] Key type .llcrypt unregistered [ 5634.054248] Key type ._llcrypt unregistered [ 5661.028880] Key type ._llcrypt registered [ 5661.033226] Key type .llcrypt registered [ 5661.140910] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5662.168074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5662.308282] alg: No test for adler32 (adler32-zlib) [ 5663.350784] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 5663.693080] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5665.385425] Key type lgssc registered [ 5666.729798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5669.967191] alg: No test for zstdfast (zstdfast-lustre-generic) [ 5669.997157] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 5675.796628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5675.814495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5677.278845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.684813] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5687.787849] LustreError: 180605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.808164] LustreError: 180605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5690.799188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.182117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5695.945709] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5702.059602] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5705.849346] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5713.075024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5713.458818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5719.592777] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5722.693381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5726.623311] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5733.904746] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 5735.907855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5735.913584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.940374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5737.957208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5737.959210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5737.965569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.993059] Lustre: Skipped 1 previous similar message [ 5739.531062] Lustre: server umount lustre-OST0000 complete [ 5745.720843] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff99a884b30800.ost_server_uuid 50 [ 5747.552606] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5763.192064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.209981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5763.216650] Lustre: Skipped 1 previous similar message [ 5767.201550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.211350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5767.214422] Lustre: Skipped 1 previous similar message [ 5769.560692] Lustre: server umount lustre-MDT0000 complete [ 5772.260252] LustreError: 180600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5772.271235] LustreError: 180600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5773.481277] LustreError: 180587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116002 with bad export cookie 4606979377986797692 [ 5773.490434] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5773.500547] LustreError: 180587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5773.847705] Lustre: server umount lustre-MDT0001 complete [ 5783.279079] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:40:10 (1789116010) [ 5790.085032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.663508] LustreError: 184087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.787103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5795.125281] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5795.811609] LustreError: 184088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.932225] LustreError: 184087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5804.280688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5804.885838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5809.068428] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5814.522205] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5818.285081] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5824.138773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5824.573734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.908357] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5835.825337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5836.877928] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5843.637145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5843.782076] Lustre: lustre-OST0001: new disk, initializing [ 5843.788206] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5843.796562] Lustre: 186588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5843.874304] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5850.222598] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5853.275813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5853.293416] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5853.383629] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5859.615156] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5863.914954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.925159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5863.943749] Lustre: Skipped 1 previous similar message [ 5863.968036] Lustre: Skipped 1 previous similar message [ 5869.192671] Lustre: server umount lustre-OST0000 complete [ 5870.565099] LustreError: 185681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5875.606604] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5875.857335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5879.652389] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5879.839381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5889.531465] LustreError: 185680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.555188] LustreError: 185680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5894.625129] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5894.632086] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.645877] LustreError: Skipped 1 previous similar message [ 5894.651958] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5894.669736] Lustre: Skipped 1 previous similar message [ 5894.685653] Lustre: Skipped 4 previous similar messages [ 5900.150528] Lustre: server umount lustre-OST0001 complete [ 5907.425025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5912.225733] Lustre: server umount lustre-MDT0000 complete [ 5917.349436] LustreError: 184068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116146 with bad export cookie 4606979377986798763 [ 5917.359966] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5917.371768] LustreError: 184068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5917.816659] Lustre: server umount lustre-MDT0001 complete [ 5945.857831] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:42:52 (1789116172) [ 5954.095709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5954.633713] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5954.677031] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5954.958240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5960.672346] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5970.428557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.561801] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5970.626990] Lustre: 189879:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5970.710680] LustreError: 189891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.741275] LustreError: 189891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5976.373607] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5982.193856] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5986.586706] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5993.635496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5993.799753] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5999.754497] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5999.778647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6006.095094] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6013.153742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6013.227922] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6019.219834] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6020.747691] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6025.023333] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6030.821655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6030.823161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6030.836569] LustreError: Skipped 1 previous similar message [ 6030.841632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6030.841643] Lustre: Skipped 5 previous similar messages [ 6030.888284] Lustre: Skipped 3 previous similar messages [ 6042.079151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6042.304320] Lustre: server umount lustre-OST0000 complete [ 6046.184214] LustreError: 191505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.537625] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6064.638225] Lustre: server umount lustre-OST0001 complete [ 6068.201078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6068.211423] Lustre: Skipped 2 previous similar messages [ 6074.476096] Lustre: server umount lustre-MDT0000 complete [ 6078.244691] LustreError: 189871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116307 with bad export cookie 4606979377986800170 [ 6078.250488] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6078.261070] LustreError: 189871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6078.629552] Lustre: server umount lustre-MDT0001 complete [ 6105.100924] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:45:31 (1789116331) [ 6123.114777] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 6133.329269] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6174.066279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.240156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6174.281881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6174.387133] Lustre: lustre-MDT0000: new disk, initializing [ 6174.452875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6174.457107] Lustre: Skipped 3 previous similar messages [ 6174.470536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6179.233325] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6191.483886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6191.618641] Lustre: 197508:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6191.707402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6191.715383] Lustre: Skipped 1 previous similar message [ 6191.843042] Lustre: lustre-MDT0001: new disk, initializing [ 6191.984135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6191.991693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6197.076179] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6205.512701] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6209.670403] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6217.168653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6217.451884] Lustre: lustre-OST0000: new disk, initializing [ 6217.455739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6217.463838] Lustre: 199250:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6224.142506] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6227.515061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6227.522219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6227.605991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6231.956340] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6236.062640] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6236.289779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6240.951304] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6241.166377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6243.299776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6243.301582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.306962] LustreError: Skipped 2 previous similar messages [ 6243.309355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6243.316542] Lustre: Skipped 2 previous similar messages [ 6243.318129] Lustre: Skipped 16 previous similar messages [ 6249.158444] Lustre: server umount lustre-OST0000 complete [ 6267.359369] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6267.671673] Lustre: server umount lustre-MDT0000 complete [ 6268.901801] LustreError: 197519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6268.939291] LustreError: 197519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6272.588158] LustreError: 197498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116501 with bad export cookie 4606979377986801962 [ 6272.593874] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6272.595917] LustreError: 197498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6272.946649] Lustre: server umount lustre-MDT0001 complete [ 6284.031407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6290.525091] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6302.096701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6302.661874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6302.677476] Lustre: Skipped 3 previous similar messages [ 6306.977661] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6312.419535] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6316.543521] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6323.845842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6331.459337] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6337.986754] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6343.371556] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6343.646050] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6348.349879] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6348.567213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6350.818036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6350.831811] LustreError: Skipped 2 previous similar messages [ 6356.734339] Lustre: server umount lustre-OST0000 complete [ 6362.760598] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6367.673108] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6396.736451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6402.486640] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6405.285311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.313361] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.314486] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6409.245093] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6416.362565] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6426.081206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6426.092557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.096981] LustreError: Skipped 1 previous similar message [ 6426.097662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6426.097666] Lustre: Skipped 13 previous similar messages [ 6426.129320] Lustre: Skipped 5 previous similar messages [ 6427.194947] Lustre: server umount lustre-OST0000 complete [ 6438.401778] Lustre: server umount lustre-MDT0000 complete [ 6442.154886] LustreError: 201303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116670 with bad export cookie 4606979377986802844 [ 6442.156669] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6442.171653] LustreError: 201303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6442.535547] Lustre: server umount lustre-MDT0001 complete [ 6450.942179] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:51:17 (1789116677) [ 6457.563203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6462.380629] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6472.550722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6477.823847] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6483.676729] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6487.532729] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6494.175271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6501.344273] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6507.591255] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6512.809481] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6513.039333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6517.803260] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6518.173805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6525.894515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.067418] Lustre: lustre-OST0001: new disk, initializing [ 6526.071188] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6526.078685] Lustre: 209771:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6531.965593] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6535.762931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6535.783198] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6535.888863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6540.686408] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6546.720927] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6547.199959] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6553.470923] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6553.811214] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6556.646623] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6556.666904] LustreError: Skipped 1 previous similar message [ 6562.157719] Lustre: server umount lustre-OST0001 complete [ 6566.882886] LustreError: 208371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6566.908443] LustreError: 208371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 6569.428870] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6569.646964] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6574.172398] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6574.375691] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6603.001427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6603.254922] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6603.261807] Lustre: Skipped 6 previous similar messages [ 6608.377434] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6608.393671] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6608.394680] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6608.419037] Lustre: Skipped 1 previous similar message [ 6610.087773] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6614.827320] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6616.973080] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6626.078302] Lustre: server umount lustre-OST0001 complete [ 6649.205231] Lustre: server umount lustre-OST0000 complete [ 6668.255147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6668.439572] Lustre: server umount lustre-MDT0000 complete [ 6672.356340] LustreError: 206754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116901 with bad export cookie 4606979377986803810 [ 6672.362118] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6672.367211] LustreError: 206754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6678.795948] Lustre: server umount lustre-MDT0001 complete [ 6708.804830] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:55:36 (1789116936) [ 6716.978534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.340310] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6717.369471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6722.901943] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6731.921519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.027027] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6732.061238] Lustre: 214728:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6736.978273] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6743.071733] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6747.163182] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6754.422681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6754.671624] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6761.176306] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6766.358419] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6775.427666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6775.505843] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6781.061145] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6787.155735] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6788.930494] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6795.217946] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6795.457969] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6799.498403] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6799.645225] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6804.823847] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99a88666c800.ost_server_uuid 50 [ 6806.908626] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99a88666c800.ost_server_uuid in IDLE state after 0 sec [ 6808.029324] Lustre: Permanently deactivating lustre-OST0001 [ 6808.048640] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6811.942679] Lustre: Permanently reactivating lustre-OST0001 [ 6818.607295] Lustre: server umount lustre-OST0001 complete [ 6820.839082] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6820.840097] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6820.848861] LustreError: Skipped 3 previous similar messages [ 6820.854770] Lustre: Skipped 10 previous similar messages [ 6825.463040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6825.470568] Lustre: Skipped 19 previous similar messages [ 6828.564791] Lustre: server umount lustre-OST0000 complete [ 6838.575572] Lustre: server umount lustre-MDT0000 complete [ 6844.147913] LustreError: 215451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117072 with bad export cookie 4606979377986805098 [ 6844.159717] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6844.164760] LustreError: 215451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6844.537694] Lustre: server umount lustre-MDT0001 complete [ 6871.690322] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:58:18 (1789117098) [ 6879.333204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6879.650152] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6879.676280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6879.687834] Lustre: Skipped 5 previous similar messages [ 6883.959882] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6892.820558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6892.977280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6893.025668] Lustre: 220838:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6898.425561] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6904.446625] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6908.934131] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6915.891507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6916.056666] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6916.073784] Lustre: Permanently deactivating lustre-OST0000 [ 6916.083602] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6923.578731] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6925.427058] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6931.547215] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6942.393358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6942.451616] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6948.592116] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6950.523357] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6954.679764] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6957.473776] Lustre: Permanently reactivating lustre-OST0000 [ 6957.517256] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6957.527549] Lustre: Skipped 2 previous similar messages [ 6964.496282] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6964.497540] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 39 seconds. I think it's dead, and I am evicting it. exp 00000000d0de12bb, cur 1789117193 deadline 1789117184 last 1789117154 [ 6964.519249] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6964.580747] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6964.685923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6978.472813] Lustre: server umount lustre-OST0001 complete [ 6989.522910] Lustre: server umount lustre-OST0000 complete [ 7003.525994] Lustre: server umount lustre-MDT0000 complete [ 7008.119122] LustreError: 220831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117236 with bad export cookie 4606979377986806778 [ 7008.135227] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7008.705311] Lustre: server umount lustre-MDT0001 complete [ 7028.790908] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7031.292550] Key type lgssc unregistered [ 7031.822694] LNet: 225274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7031.830599] LNetError: 225274:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7032.871968] LNet: Removed LNI 192.168.204.144@tcp [ 7033.797322] Key type .llcrypt unregistered [ 7033.799745] Key type ._llcrypt unregistered [ 7060.466068] Key type ._llcrypt registered [ 7060.468071] Key type .llcrypt registered [ 7060.617955] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7061.346477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7061.393932] alg: No test for adler32 (adler32-zlib) [ 7062.654792] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 7062.923424] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7064.671280] Key type lgssc registered [ 7065.921709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7069.274394] alg: No test for zstdfast (zstdfast-lustre-generic) [ 7069.287428] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 7078.334949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7078.411643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7080.286069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7084.740488] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7089.636460] LustreError: 226828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7089.688520] LustreError: 226828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7090.666171] LustreError: 226828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7095.785579] LustreError: 226829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7096.287487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7096.524599] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7096.594488] Lustre: 226816:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7096.620550] Lustre: Permanently deactivating lustre-MDT0001 [ 7096.651568] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7097.162279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7098.080477] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7098.085901] LustreError: 226859:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7098.096584] LustreError: 226859:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7098.110730] LustreError: 226859:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7102.590059] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7109.127673] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7112.775169] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7120.396049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7121.109634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.153355] LustreError: 228443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7125.248207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7129.194769] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7130.599150] LustreError: 228442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7130.643944] LustreError: 228442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7137.118408] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7140.845658] LustreError: 228518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.869702] LustreError: 228518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7145.204045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7145.445196] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7150.588765] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7150.607078] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7151.524360] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7157.521601] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7161.008284] Lustre: Permanently reactivating lustre-MDT0001 [ 7161.022604] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7161.025195] Lustre: Skipped 2 previous similar messages [ 7168.001534] Lustre: lustre-MDT0001: Client 91e94892-34c8-49e0-bd84-d3e4863b752d (at 192.168.204.44@tcp) reconnecting [ 7174.202547] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7176.324914] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7182.970542] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7183.250523] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7188.936770] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7189.237374] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7192.623308] Lustre: Permanently deactivating lustre-MDT0001 [ 7192.660578] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7192.669121] Lustre: Skipped 2 previous similar messages [ 7198.349317] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7212.020291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.032719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7212.042179] Lustre: Skipped 2 previous similar messages [ 7212.064516] Lustre: Skipped 3 previous similar messages [ 7216.902570] Lustre: server umount lustre-MDT0000 complete [ 7217.130355] LustreError: 226829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7221.306846] LustreError: 226809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117450 with bad export cookie 6599879756675674508 [ 7221.315334] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7221.328943] LustreError: 226809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7221.690911] Lustre: server umount lustre-MDT0001 complete [ 7227.112261] Lustre: server umount lustre-OST0000 complete [ 7242.176657] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:04:28 (1789117468) [ 7255.416810] Lustre: server umount lustre-OST0001 complete [ 7263.590069] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7274.318839] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7317.452762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7317.782898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7317.793240] Lustre: Skipped 2 previous similar messages [ 7317.819711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7317.870692] Lustre: lustre-MDT0000: new disk, initializing [ 7317.925516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.941217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7322.513059] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7335.537780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.750365] Lustre: 234505:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7335.850440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7335.857575] Lustre: Skipped 1 previous similar message [ 7336.010266] Lustre: lustre-MDT0001: new disk, initializing [ 7336.105151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7336.139948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7336.166431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7342.134108] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7351.350935] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7357.026392] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7365.051614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7365.357578] Lustre: lustre-OST0000: new disk, initializing [ 7365.365652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7365.373732] Lustre: 236253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7365.515156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7372.824144] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7374.423434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7374.437229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7374.562206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7380.973884] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7383.112928] LustreError: 235635:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7393.208162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7393.218308] LustreError: 235635:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7393.245509] LustreError: 234511:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7393.416739] Lustre: lustre-OST0001: new disk, initializing [ 7393.427508] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7393.446343] Lustre: 237254:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7400.038816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7400.050543] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7400.115370] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7401.591808] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7403.288074] LustreError: 234511:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7410.379922] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7412.703914] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7415.776948] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7415.782318] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7415.793501] LustreError: Skipped 1 previous similar message [ 7415.805878] Lustre: Skipped 1 previous similar message [ 7415.819218] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7420.899867] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7420.909197] Lustre: Skipped 2 previous similar messages [ 7421.269540] Lustre: server umount lustre-OST0001 complete [ 7426.016603] LustreError: 236236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.066138] LustreError: 236236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7441.377182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7441.378943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.395104] LustreError: Skipped 1 previous similar message [ 7441.399891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7441.426213] Lustre: Skipped 2 previous similar messages [ 7441.451894] Lustre: Skipped 1 previous similar message [ 7446.129616] Lustre: server umount lustre-OST0000 complete [ 7451.618899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7451.622071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7451.654358] Lustre: Skipped 1 previous similar message [ 7457.309507] Lustre: server umount lustre-MDT0000 complete [ 7461.815414] LustreError: 234497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117690 with bad export cookie 6599879756675677105 [ 7461.826321] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7461.831965] LustreError: 234497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7476.191119] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7476.482376] Lustre: server umount lustre-MDT0001 complete [ 7490.369355] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7493.329508] Key type lgssc unregistered [ 7493.692076] LNet: 239158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7493.701923] LNetError: 239158:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7493.736166] LNet: Removed LNI 192.168.204.144@tcp [ 7494.606601] Key type .llcrypt unregistered [ 7494.611860] Key type ._llcrypt unregistered [ 7540.104271] Key type ._llcrypt registered [ 7540.105825] Key type .llcrypt registered [ 7540.260395] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7541.836915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7542.029664] alg: No test for adler32 (adler32-zlib) [ 7543.187864] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 7543.491768] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7545.183992] Key type lgssc registered [ 7546.938693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7549.661376] alg: No test for zstdfast (zstdfast-lustre-generic) [ 7549.682716] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 7555.433401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7555.455366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7556.832587] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7556.850935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7557.094365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.474375] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7570.157829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7570.333290] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7570.372490] Lustre: 241433:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7570.662516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7575.086440] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7580.424992] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7584.799564] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7591.603114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7591.834055] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7592.066847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.367419] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7598.244677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7603.837768] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7659.488606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7659.513751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7659.537103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7660.001572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7661.402610] Lustre: server umount lustre-OST0000 complete [ 7665.883969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7680.275234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7680.961682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7687.298866] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7692.793456] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7699.307868] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7699.333086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7699.335680] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7699.367059] Lustre: Skipped 1 previous similar message [ 7735.264640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7735.277331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.282498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7735.284137] Lustre: Skipped 2 previous similar messages [ 7735.300984] Lustre: Skipped 2 previous similar messages [ 7737.156721] Lustre: server umount lustre-OST0000 complete [ 7744.992830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7745.008214] LustreError: Skipped 1 previous similar message [ 7745.017308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.045432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7750.627293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7750.645130] Lustre: Skipped 3 previous similar messages [ 7755.231141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7755.459259] Lustre: server umount lustre-MDT0000 complete [ 7759.841181] LustreError: 241424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117988 with bad export cookie 528232354347978220 [ 7759.845995] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7759.858286] LustreError: 241424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7760.398627] Lustre: server umount lustre-MDT0001 complete [ 7773.085478] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7776.139572] Key type lgssc unregistered [ 7776.520966] LNet: 246387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7776.530611] LNetError: 246387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7776.552956] LNet: Removed LNI 192.168.204.144@tcp [ 7777.434883] Key type .llcrypt unregistered [ 7777.436529] Key type ._llcrypt unregistered [ 7804.630366] Key type ._llcrypt registered [ 7804.632629] Key type .llcrypt registered [ 7804.765767] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7806.099173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7806.147593] alg: No test for adler32 (adler32-zlib) [ 7807.192176] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 7807.393987] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7809.065443] Key type lgssc registered [ 7810.384470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7812.950452] alg: No test for zstdfast (zstdfast-lustre-generic) [ 7812.974307] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 7818.221000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7818.244898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.798580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7823.807437] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7829.988486] LustreError: 247872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7829.999987] LustreError: 247872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7834.695041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.039748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7840.360300] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7847.634229] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7852.848685] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7860.514681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7861.104355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7865.209368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7868.999337] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7875.511817] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7905.258457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7905.260782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7905.278835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7905.288800] Lustre: Skipped 1 previous similar message [ 7909.426701] Lustre: server umount lustre-OST0000 complete [ 7915.490536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7915.509514] Lustre: Skipped 1 previous similar message [ 7915.513967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7915.522021] Lustre: Skipped 1 previous similar message [ 7919.728707] Lustre: server umount lustre-MDT0000 complete [ 7920.613079] LustreError: 247872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.638828] LustreError: 247872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7924.086688] LustreError: 247852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118152 with bad export cookie 16760632907089460566 [ 7924.094018] LustreError: 247852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7924.095908] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7924.443938] Lustre: server umount lustre-MDT0001 complete [ 7936.371474] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7939.764842] Key type lgssc unregistered [ 7940.082885] LNet: 251824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7940.087721] LNetError: 251824:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7940.109521] LNet: Removed LNI 192.168.204.144@tcp [ 7940.980606] Key type .llcrypt unregistered [ 7940.982214] Key type ._llcrypt unregistered [ 7961.491905] Key type ._llcrypt registered [ 7961.495929] Key type .llcrypt registered [ 7961.603732] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7962.506945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7962.659054] alg: No test for adler32 (adler32-zlib) [ 7963.949882] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 7964.295293] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7966.034302] Key type lgssc registered [ 7967.484693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7971.221896] alg: No test for zstdfast (zstdfast-lustre-generic) [ 7971.228488] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 7977.233736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7977.250527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7978.697453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7983.297427] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7989.224039] LustreError: 253112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7989.249334] LustreError: 253112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7992.182323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7992.525610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7996.697944] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8002.528315] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8006.574224] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8013.433234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8013.760159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8019.944238] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8025.016234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8025.942688] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8038.757502] Lustre: server umount lustre-OST0000 complete [ 8044.006386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8044.031517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.063668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.537750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.543456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.571435] Lustre: Skipped 1 previous similar message [ 8050.660085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8050.673294] Lustre: Skipped 1 previous similar message [ 8052.872580] Lustre: server umount lustre-MDT0000 complete [ 8055.776345] LustreError: 253111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8055.803819] LustreError: 253111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8056.739165] Lustre: 252370:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789118269/real 1789118269] req@000000001561b98d x1876026398370688/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789118285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8056.803013] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.835404] LustreError: 253112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8056.850350] LustreError: 253112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8056.951913] LustreError: 253092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118285 with bad export cookie 16525538951484182762 [ 8056.956452] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8056.960558] LustreError: 253092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8057.203594] Lustre: server umount lustre-MDT0001 complete [ 8070.347145] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8073.456258] Key type lgssc unregistered [ 8073.746705] LNet: 256432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8073.757512] LNetError: 256432:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8074.790382] LNet: Removed LNI 192.168.204.144@tcp [ 8075.810208] Key type .llcrypt unregistered [ 8075.816105] Key type ._llcrypt unregistered [ 8102.946135] Key type ._llcrypt registered [ 8102.948252] Key type .llcrypt registered [ 8103.072837] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8104.554183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8104.672046] alg: No test for adler32 (adler32-zlib) [ 8105.676777] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 8106.031120] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8107.791218] Key type lgssc registered [ 8109.437549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8113.057922] alg: No test for zstdfast (zstdfast-lustre-generic) [ 8113.066818] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 8119.416883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8119.455221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8121.015530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8125.945650] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8130.552111] LustreError: 257919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8130.581670] LustreError: 257919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8135.531434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8135.651745] LustreError: 257918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8136.198262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8141.397160] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8147.513460] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8151.696240] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8158.758758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8159.256195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8165.425774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8166.375675] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8172.608218] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8201.187905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8201.197201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8201.199441] Lustre: Skipped 1 previous similar message [ 8201.201193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8201.215744] Lustre: Skipped 1 previous similar message [ 8203.036453] Lustre: server umount lustre-OST0000 complete [ 8207.840540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8207.842979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8207.866865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8211.435966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8211.443920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8213.454968] Lustre: server umount lustre-MDT0000 complete [ 8216.546044] LustreError: 257918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8216.569631] LustreError: 257918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8217.350862] LustreError: 259537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118446 with bad export cookie 5764965482000732977 [ 8217.361274] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8217.364738] LustreError: 259537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8217.684117] Lustre: server umount lustre-MDT0001 complete [ 8229.904922] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8232.867828] Key type lgssc unregistered [ 8233.183714] LNet: 261869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8233.194712] LNetError: 261869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8233.211744] LNet: Removed LNI 192.168.204.144@tcp [ 8234.055493] Key type .llcrypt unregistered [ 8234.060758] Key type ._llcrypt unregistered [ 8252.646430] Key type ._llcrypt registered [ 8252.655242] Key type .llcrypt registered [ 8252.785634] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8254.008338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8254.131225] alg: No test for adler32 (adler32-zlib) [ 8255.338566] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 8255.894063] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8257.695957] Key type lgssc registered [ 8259.684880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8264.575420] alg: No test for zstdfast (zstdfast-lustre-generic) [ 8264.584700] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 8271.293514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8271.348393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8273.136888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8277.562509] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8282.609790] LustreError: 263172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8282.637469] LustreError: 263172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8287.717390] LustreError: 263174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8288.243694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.864262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8293.135661] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8298.485130] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8302.596667] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8310.518303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8310.888770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8316.528756] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8318.062685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8321.982834] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8334.819480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8334.827199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.850080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8338.916431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8338.921873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8338.944919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8344.036317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8344.052219] Lustre: Skipped 2 previous similar messages [ 8348.127380] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8348.281817] Lustre: server umount lustre-OST0000 complete [ 8354.273994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.284366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8354.287509] Lustre: Skipped 1 previous similar message [ 8358.193701] Lustre: server umount lustre-MDT0000 complete [ 8359.397824] LustreError: 263184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8359.431460] LustreError: 263184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8361.985453] LustreError: 263138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118590 with bad export cookie 10752250108541097961 [ 8361.995536] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8362.003314] LustreError: 263138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8362.276074] Lustre: server umount lustre-MDT0001 complete [ 8373.120580] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8376.182887] Key type lgssc unregistered [ 8376.636214] LNet: 266519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8376.652632] LNetError: 266519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8377.704497] LNet: Removed LNI 192.168.204.144@tcp [ 8378.827212] Key type .llcrypt unregistered [ 8378.829458] Key type ._llcrypt unregistered [ 8392.577819] Key type ._llcrypt registered [ 8392.580623] Key type .llcrypt registered [ 8392.651778] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8409.853282] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8425.797989] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8426.940876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8427.097038] alg: No test for adler32 (adler32-zlib) [ 8428.165038] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 8428.458416] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8430.167177] Key type lgssc registered [ 8431.385332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8434.306781] alg: No test for zstdfast (zstdfast-lustre-generic) [ 8434.320509] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 8465.268558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8465.299722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.537810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8466.565880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8466.669965] Lustre: lustre-MDT0000: new disk, initializing [ 8466.810683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8466.843043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8470.871219] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8481.077970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8481.150921] Lustre: 270108:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8481.191478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8481.195448] Lustre: Skipped 1 previous similar message [ 8481.280410] Lustre: lustre-MDT0001: new disk, initializing [ 8481.368361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8481.412351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8481.424102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8484.599604] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8490.214396] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8493.655989] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8500.199856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8500.488992] Lustre: lustre-OST0000: new disk, initializing [ 8500.495612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8500.510626] Lustre: 271849:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8500.609296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8505.977655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8505.991918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8506.111607] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8506.145419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8513.056501] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8518.310878] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8518.571467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8523.080762] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8523.329395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8526.819664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8526.820255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.826013] LustreError: Skipped 1 previous similar message [ 8526.827198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8526.851982] Lustre: Skipped 1 previous similar message [ 8531.297869] Lustre: server umount lustre-OST0000 complete [ 8537.057112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8537.058566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8537.065181] Lustre: Skipped 1 previous similar message [ 8537.090778] Lustre: Skipped 2 previous similar messages [ 8542.176899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8542.180923] Lustre: Skipped 1 previous similar message [ 8546.276845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8546.285630] Lustre: Skipped 1 previous similar message [ 8549.343348] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8549.513334] Lustre: server umount lustre-MDT0000 complete [ 8552.876296] LustreError: 270100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118781 with bad export cookie 3662910193021611437 [ 8552.878685] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8552.898369] LustreError: 270100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8553.266054] Lustre: server umount lustre-MDT0001 complete [ 8561.236440] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:26:28 (1789118788) [ 8567.316308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8567.757597] LustreError: 274112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8567.779962] LustreError: 274112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8567.843794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8571.774981] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8572.899411] LustreError: 274113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8578.017417] LustreError: 274112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8579.772789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8580.378787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8584.395572] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8589.474324] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8592.774669] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8598.436096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8598.735205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8603.528835] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8608.356739] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8621.538796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8621.552901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8621.577625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8623.587622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8623.595676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8627.420325] Lustre: server umount lustre-OST0000 complete [ 8631.776913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8631.789229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8631.805559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8631.809689] Lustre: Skipped 2 previous similar messages [ 8636.990770] Lustre: server umount lustre-MDT0000 complete [ 8638.956524] LustreError: 274109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8638.983237] LustreError: 274109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8640.412338] LustreError: 274092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118869 with bad export cookie 3662910193021612319 [ 8640.416945] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8640.424183] LustreError: 274092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8640.734933] Lustre: server umount lustre-MDT0001 complete [ 8649.981395] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8652.063890] Key type lgssc unregistered [ 8652.301343] LNet: 277311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8652.312206] LNetError: 277311:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8652.337968] LNet: Removed LNI 192.168.204.144@tcp [ 8652.974794] Key type .llcrypt unregistered [ 8652.976996] Key type ._llcrypt unregistered [ 8674.605751] Key type ._llcrypt registered [ 8674.607355] Key type .llcrypt registered [ 8674.708961] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8689.861766] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8690.799272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8690.807909] alg: No test for adler32 (adler32-zlib) [ 8692.005552] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 8692.277524] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8694.007141] Key type lgssc registered [ 8695.230753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8698.357068] alg: No test for zstdfast (zstdfast-lustre-generic) [ 8698.370967] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 8734.679667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8734.698389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8735.900664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8735.938877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8736.003829] Lustre: lustre-MDT0000: new disk, initializing [ 8736.080531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.093914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8739.380432] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8748.542343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.663598] Lustre: 281186:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8748.730338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8748.740813] Lustre: Skipped 1 previous similar message [ 8748.853883] Lustre: lustre-MDT0001: new disk, initializing [ 8748.959359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8748.996937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8749.017924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8752.977795] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8760.244531] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8764.183070] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8770.793089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8771.088665] Lustre: lustre-OST2710: new disk, initializing [ 8771.092922] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8771.105462] Lustre: 282924:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8771.212455] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8777.410113] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8778.325654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8778.335302] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8778.470571] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8785.148979] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8792.089778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8792.357125] Lustre: lustre-OST03e8: new disk, initializing [ 8792.364567] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8792.375445] Lustre: 283877:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8792.538780] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8798.360389] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8799.382525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8799.402036] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8799.535131] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8803.906961] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8809.990327] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8810.120622] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8813.640210] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8813.824222] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8818.022202] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8818.296473] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8822.886873] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8823.183699] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8830.440399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8830.468989] Lustre: Skipped 1 previous similar message [ 8830.589692] Lustre: server umount lustre-MDT0000 complete [ 8834.054458] LustreError: 281178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119062 with bad export cookie 11372449583882703087 [ 8834.065504] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8834.070223] LustreError: 281178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8834.305477] Lustre: server umount lustre-MDT0001 complete [ 8848.257198] Lustre: server umount lustre-OST2710 complete [ 8861.873285] Lustre: server umount lustre-OST03e8 complete [ 8879.103952] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8889.362872] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8928.825664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8929.024987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8929.044536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8929.116927] Lustre: lustre-MDT0000: new disk, initializing [ 8929.204723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8929.234596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8932.705799] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8942.948024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8943.107284] Lustre: 288923:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8943.230678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8943.236484] Lustre: Skipped 1 previous similar message [ 8943.320763] Lustre: lustre-MDT0001: new disk, initializing [ 8943.418518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8943.445443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8943.460790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8947.939815] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8955.738714] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8959.980576] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8966.954162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8967.355886] Lustre: lustre-OST0000: new disk, initializing [ 8967.359684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8967.365791] Lustre: 290664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8967.444823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8973.737346] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8975.396528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8975.409316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8975.510523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8981.559605] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8986.748515] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8986.966920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8991.444848] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8991.702820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8996.329323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8996.334261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8996.338915] LustreError: Skipped 1 previous similar message [ 8996.340910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8996.363445] Lustre: Skipped 3 previous similar messages [ 8996.383625] Lustre: Skipped 1 previous similar message [ 8999.966219] Lustre: server umount lustre-OST0000 complete [ 9005.024308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9005.031415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9005.052267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9006.560040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9006.576731] Lustre: Skipped 1 previous similar message [ 9011.680587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9011.683481] Lustre: Skipped 1 previous similar message [ 9016.803127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9016.815494] Lustre: Skipped 1 previous similar message [ 9019.359188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9019.495211] Lustre: server umount lustre-MDT0000 complete [ 9021.922449] LustreError: 288934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9021.950736] LustreError: 288934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9023.476212] LustreError: 290663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119252 with bad export cookie 11372449583882704767 [ 9023.483983] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9023.485703] LustreError: 290663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9023.865785] Lustre: server umount lustre-MDT0001 complete [ 9033.088671] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:34:20 (1789119260) [ 9034.762037] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9036.572381] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:34:23 (1789119263) [ 9040.677122] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9068.878536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9069.095269] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9069.110259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9069.321408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9074.606256] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9083.182129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9083.290758] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9083.333665] Lustre: 294372:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9088.172868] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9109.895843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9110.291563] Lustre: Denying initial registration attempt from nid 192.168.204.144@tcp, specified as failover [ 9110.307408] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9110.318111] LustreError: 295790:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 9110.329774] LustreError: 295790:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9110.339537] LustreError: 295790:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9110.477950] Lustre: server umount lustre-OST0000 complete [ 9110.481449] LustreError: 295790:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9119.717212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9119.724139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9119.735269] Lustre: Skipped 1 previous similar message [ 9119.743069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9125.857479] LustreError: 294379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9125.872641] LustreError: 294379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9127.839404] Lustre: server umount lustre-MDT0000 complete [ 9130.978906] LustreError: 294384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9130.997335] LustreError: 294384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9131.193142] LustreError: 294365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119359 with bad export cookie 11372449583882705649 [ 9131.209677] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9131.211345] LustreError: 294365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9131.604529] Lustre: server umount lustre-MDT0001 complete [ 9142.395361] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9144.839804] Key type lgssc unregistered [ 9145.150592] LNet: 296895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9145.162545] LNetError: 296895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9145.185948] LNet: Removed LNI 192.168.204.144@tcp [ 9145.917252] Key type .llcrypt unregistered [ 9145.921299] Key type ._llcrypt unregistered [ 9156.118158] Key type ._llcrypt registered [ 9156.120193] Key type .llcrypt registered [ 9156.221122] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9157.211085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9157.259107] alg: No test for adler32 (adler32-zlib) [ 9158.380851] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 9158.716163] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9160.495155] Key type lgssc registered [ 9161.606922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9164.631136] alg: No test for zstdfast (zstdfast-lustre-generic) [ 9164.658441] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 9209.981764] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9218.072918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9218.095719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9219.397811] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9219.423664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9219.651334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9223.594330] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9231.375565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9231.433172] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9231.462776] Lustre: 299723:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9231.711822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9235.761831] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9241.071236] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9244.583923] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9263.599688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9263.914022] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9264.165153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9270.745957] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9276.397370] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9278.944311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9278.948778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9278.963272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9280.747941] Lustre: server umount lustre-OST0000 complete [ 9288.160778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9288.166113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9288.181749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9289.185434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9289.187948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9289.231872] Lustre: Skipped 1 previous similar message [ 9291.194740] Lustre: server umount lustre-MDT0000 complete [ 9294.527973] LustreError: 299716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119523 with bad export cookie 6830792965673199551 [ 9294.533392] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9294.538668] LustreError: 299716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9305.207386] Lustre: server umount lustre-MDT0001 complete [ 9314.687097] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9317.711333] Key type lgssc unregistered [ 9317.982891] LNet: 302937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9317.989486] LNetError: 302937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9318.002562] LNet: Removed LNI 192.168.204.144@tcp [ 9318.832156] Key type .llcrypt unregistered [ 9318.837843] Key type ._llcrypt unregistered [ 9344.126500] Key type ._llcrypt registered [ 9344.132700] Key type .llcrypt registered [ 9344.290725] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9345.546762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9345.606471] alg: No test for adler32 (adler32-zlib) [ 9346.761356] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 9347.145576] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9348.968648] Key type lgssc registered [ 9350.261201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9353.417905] alg: No test for zstdfast (zstdfast-lustre-generic) [ 9353.432561] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 9359.524770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9359.553268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9361.045669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.818573] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9370.600110] LustreError: 304422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.638162] LustreError: 304422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9374.443649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9374.821796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9379.288306] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9384.088976] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9387.678321] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9397.427819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9398.005051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9404.859582] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9410.705434] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9420.769770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9420.771082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9420.782285] Lustre: Skipped 2 previous similar messages [ 9420.783368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9420.809647] Lustre: Skipped 2 previous similar messages [ 9423.894924] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 9425.759843] Lustre: server umount lustre-MDT0000 complete [ 9425.891052] LustreError: 304419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9425.909643] LustreError: 304419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9428.955771] LustreError: 304417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9429.402540] LustreError: 304402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119658 with bad export cookie 7900099984900278750 [ 9429.407263] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9429.417430] LustreError: 304402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9431.008830] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9431.015082] LustreError: 304419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9431.024717] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9434.023772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9435.937590] Lustre: server umount lustre-MDT0001 complete [ 9443.771771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9452.566365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9455.520331] LustreError: 303678:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000777366a2 x1876027847513600/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9455.887330] LustreError: 306030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9455.907819] LustreError: 306030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9456.010674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9456.056570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9459.780174] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9465.191508] LustreError: 306031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9465.199150] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9465.210345] LustreError: 306031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9468.368576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9468.610326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9473.941078] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9474.060695] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9480.199641] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9481.867653] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9486.166612] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9487.520652] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9491.937791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9491.948109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9491.957853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9491.964627] Lustre: Skipped 1 previous similar message [ 9494.506182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9494.512262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9504.739178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9504.745355] Lustre: Skipped 5 previous similar messages [ 9506.271113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9506.561724] Lustre: server umount lustre-OST0000 complete [ 9514.983619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9514.983628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9516.981099] Lustre: server umount lustre-MDT0000 complete [ 9520.098568] LustreError: 307682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9520.128271] LustreError: 307682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9520.715507] LustreError: 306042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119749 with bad export cookie 7900099984900286373 [ 9520.717518] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9520.723267] LustreError: 306042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9520.991392] Lustre: server umount lustre-MDT0001 complete [ 9532.282698] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9534.905033] Key type lgssc unregistered [ 9535.258848] LNet: 310109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9535.271451] LNetError: 310109:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9536.307813] LNet: Removed LNI 192.168.204.144@tcp [ 9537.183418] Key type .llcrypt unregistered [ 9537.185968] Key type ._llcrypt unregistered [ 9560.588186] Key type ._llcrypt registered [ 9560.591398] Key type .llcrypt registered [ 9560.689344] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9561.665598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9561.772876] alg: No test for adler32 (adler32-zlib) [ 9562.838317] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 9563.113565] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9564.823480] Key type lgssc registered [ 9565.983469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9568.968560] alg: No test for zstdfast (zstdfast-lustre-generic) [ 9568.984720] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 9574.259523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9574.295868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9575.971846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.225162] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9586.144979] LustreError: 311593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9586.167085] LustreError: 311593:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9590.214781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9590.644145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9595.708134] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9601.785287] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9606.561747] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9611.233632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9611.240795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9611.241480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9615.399097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9615.408232] Lustre: Skipped 2 previous similar messages [ 9615.675794] Lustre: server umount lustre-MDT0000 complete [ 9619.243487] LustreError: 311575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119848 with bad export cookie 15730757961223572741 [ 9619.248612] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9619.249211] LustreError: 311575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9619.580193] Lustre: server umount lustre-MDT0001 complete [ 9632.873903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9633.175913] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9633.201366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9633.369688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9637.014126] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9645.353507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9645.452631] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9645.484500] Lustre: 313953:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9645.716538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9650.120812] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9656.393155] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9660.280233] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9671.346026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9671.552494] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9671.559111] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9671.568218] LustreError: 315610:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 9671.571945] LustreError: 315610:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9671.578570] LustreError: 315610:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9671.623860] Lustre: server umount lustre-OST0000 complete [ 9671.625922] LustreError: 315610:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9682.180588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9682.444918] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9682.738322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9688.109343] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9690.225224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9691.965872] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9697.943658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.024322] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9698.097161] Lustre: lustre-OST0001: new disk, initializing [ 9698.100580] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9698.106740] Lustre: 317040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9698.184945] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9703.746902] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9708.175850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9708.188374] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9708.298193] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9709.205592] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9713.632232] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9713.644237] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9713.648282] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9713.667303] Lustre: Skipped 2 previous similar messages [ 9718.134314] Lustre: server umount lustre-OST0001 complete [ 9718.761032] LustreError: 316076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9718.784048] LustreError: 316076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9723.875422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9723.879128] LustreError: 317547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9723.879898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9723.879904] Lustre: Skipped 1 previous similar message [ 9723.904873] Lustre: Skipped 1 previous similar message [ 9723.921785] LustreError: 317547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9727.674112] Lustre: server umount lustre-OST0000 complete [ 9733.088755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9733.095578] LustreError: Skipped 1 previous similar message [ 9733.100150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9733.129805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9733.139237] Lustre: Skipped 1 previous similar message [ 9737.656542] Lustre: server umount lustre-MDT0000 complete [ 9739.236556] LustreError: 313963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9739.260848] LustreError: 313963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9741.848963] LustreError: 313943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119970 with bad export cookie 15730757961223573301 [ 9741.851331] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9741.858350] LustreError: 313943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9742.300775] Lustre: server umount lustre-MDT0001 complete [ 9752.591501] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9756.766200] Key type lgssc unregistered [ 9757.173811] LNet: 318869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9757.183682] LNetError: 318869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9757.205042] LNet: Removed LNI 192.168.204.144@tcp [ 9758.467764] Key type .llcrypt unregistered [ 9758.472428] Key type ._llcrypt unregistered [ 9829.687803] Key type ._llcrypt registered [ 9829.689677] Key type .llcrypt registered [ 9829.827495] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 9845.901150] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9846.695945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9846.716596] alg: No test for adler32 (adler32-zlib) [ 9847.760181] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 9848.046304] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9849.767153] Key type lgssc registered [ 9851.099268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9854.749180] alg: No test for zstdfast (zstdfast-lustre-generic) [ 9854.758976] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 9886.811839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9886.840757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9888.205505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9888.239976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9888.353439] Lustre: lustre-MDT0000: new disk, initializing [ 9888.498348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9888.546212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9893.186997] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9904.190936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9904.328497] Lustre: 323853:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9904.401296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9904.408082] Lustre: Skipped 1 previous similar message [ 9904.487509] Lustre: lustre-MDT0001: new disk, initializing [ 9904.576366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9904.608032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9904.619047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9909.129988] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9917.305113] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9921.031224] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9927.460209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9927.776085] Lustre: lustre-OST0000: new disk, initializing [ 9927.780195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9927.786948] Lustre: 325592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9927.893188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9934.261393] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9935.924027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9935.956908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9936.125451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9941.827386] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9946.568109] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9946.774992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9951.866938] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9952.159982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9956.838270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9956.839104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9956.848428] LustreError: Skipped 1 previous similar message [ 9956.850341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9956.861187] Lustre: Skipped 1 previous similar message [ 9956.877397] Lustre: Skipped 1 previous similar message [ 9960.372509] Lustre: server umount lustre-OST0000 complete [ 9966.048559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9966.063429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9966.087287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9967.083516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9967.101510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9967.103739] Lustre: Skipped 1 previous similar message [ 9970.599607] Lustre: server umount lustre-MDT0000 complete [ 9974.459837] LustreError: 325591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120203 with bad export cookie 2069300772118645329 [ 9974.462705] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9974.475108] LustreError: 325591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9974.901408] Lustre: server umount lustre-MDT0001 complete [ 9982.768514] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:50:09 (1789120209) [ 9990.274583] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:50:17 (1789120217) [ 9999.928058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10000.376036] LustreError: 328201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10000.402276] LustreError: 328201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10000.454542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10005.118506] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10005.479223] LustreError: 328202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10010.606442] LustreError: 328202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10014.936643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10015.545296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10021.224809] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10027.023974] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10031.035827] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10038.069902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10038.352906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10043.704473] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10050.542516] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10059.366231] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10061.244724] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10063.101931] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10065.057135] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10067.147469] Lustre: Failing over lustre-MDT0000 [10067.508581] Lustre: server umount lustre-MDT0000 complete [10067.970489] LustreError: 328198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10071.522790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10071.548098] Lustre: Skipped 2 previous similar messages [10073.051428] LustreError: 328196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10073.081405] LustreError: 328196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10081.762622] LustreError: 328196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10081.793333] LustreError: 328196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10086.789766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10086.970604] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10087.264466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10087.330291] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10089.437991] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10091.593402] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10092.526666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10092.561894] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10092.628544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10098.776123] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10100.393763] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10103.998711] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10106.094164] Lustre: Failing over lustre-MDT0000 [10106.648131] Lustre: server umount lustre-MDT0000 complete [10107.872116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10107.876453] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.880926] LustreError: 329814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10107.880938] LustreError: 329814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10107.942713] Lustre: Skipped 2 previous similar messages [10123.602553] Lustre: 322025:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789120336/real 1789120336] req@00000000b1ee74d1 x1876028372897152/t0(0) o400->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789120352 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10123.633062] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10125.793860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10134.012203] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1cb7a216af92d4e6 [10134.034127] Lustre: MGC192.168.204.144@tcp: Connection restored to 0@lo (at 0@lo) [10134.044590] Lustre: Skipped 2 previous similar messages [10134.351649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10134.396508] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10134.499245] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10138.884211] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10139.627722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10139.667918] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10139.707637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10146.892625] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10148.863574] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10154.978083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10154.980033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10155.000047] Lustre: Skipped 2 previous similar messages [10155.017345] Lustre: Skipped 2 previous similar messages [10160.103457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10160.115879] Lustre: Skipped 2 previous similar messages [10161.202395] Lustre: server umount lustre-MDT0000 complete [10164.946725] LustreError: 328182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120393 with bad export cookie 2069300772118648038 [10164.971917] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10164.977555] LustreError: 328182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10165.192901] Lustre: server umount lustre-MDT0001 complete [10178.544441] Lustre: server umount lustre-OST0000 complete [10195.393309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10196.155629] LustreError: 333977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.184697] LustreError: 333977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [10201.380742] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10211.772578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10212.061865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10212.067193] Lustre: Skipped 1 previous similar message [10216.402312] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10223.284463] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10227.062077] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10235.129284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10241.528207] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10241.650227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10247.809379] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10256.869128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10256.885657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10256.920658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10260.279692] Lustre: server umount lustre-OST0000 complete [10277.859780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10277.870603] Lustre: Skipped 7 previous similar messages [10279.391385] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10279.731885] Lustre: server umount lustre-MDT0000 complete [10282.981494] LustreError: 333973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10283.006896] LustreError: 333973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10283.753091] LustreError: 333957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120512 with bad export cookie 2069300772118648787 [10283.764786] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10284.219094] Lustre: server umount lustre-MDT0001 complete [10295.812342] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [10298.716094] Key type lgssc unregistered [10299.055053] LNet: 337156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10299.072659] LNetError: 337156:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10300.133955] LNet: Removed LNI 192.168.204.144@tcp [10301.039400] Key type .llcrypt unregistered [10301.042811] Key type ._llcrypt unregistered [10329.456361] Key type ._llcrypt registered [10329.458492] Key type .llcrypt registered [10329.576907] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10330.962541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10331.008838] alg: No test for adler32 (adler32-zlib) [10332.168910] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [10332.481769] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [10334.263263] Key type lgssc registered [10336.029251] Lustre: Echo OBD driver; http://www.lustre.org/ [10339.631623] alg: No test for zstdfast (zstdfast-lustre-generic) [10339.641598] alg: No test for zstdfast (zstdfast-lustre-scomp) [10346.015636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10346.045852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10347.553834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10352.629960] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10357.732054] LustreError: 338740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10357.769954] LustreError: 338740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10362.855859] LustreError: 338741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10363.747159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10364.333983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10370.260507] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10377.567323] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10382.549863] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10391.269261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10391.763127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10392.834972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10399.563904] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10405.848250] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10419.550109] Lustre: server umount lustre-MDT0000 complete [10420.716510] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10420.719167] LustreError: 338737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10420.734643] Lustre: Skipped 1 previous similar message [10420.747929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10423.798520] LustreError: 338720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120652 with bad export cookie 16962328938799690668 [10423.814868] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10424.165229] Lustre: server umount lustre-MDT0001 complete [10439.126493] Lustre: server umount lustre-OST0000 complete [10453.713275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10454.169817] LustreError: 342111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10454.186157] LustreError: 342111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10454.260281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10459.620678] LustreError: 342112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.903996] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10468.833529] LustreError: 342112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10468.873660] LustreError: 342112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10470.040707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10470.514902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10475.202342] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10481.889732] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10486.783747] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10495.114103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10495.696054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10498.749423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10503.354465] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10508.972718] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10521.568193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10521.576712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10521.592699] Lustre: Skipped 1 previous similar message [10521.606319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10524.644422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10524.657029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10524.691787] Lustre: Skipped 1 previous similar message [10527.759581] Lustre: server umount lustre-MDT0000 complete [10529.761591] LustreError: 342106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10529.792626] LustreError: 342106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10531.751270] LustreError: 342092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120760 with bad export cookie 16962328938799691781 [10531.756357] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10531.775448] LustreError: 342092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10532.216947] Lustre: server umount lustre-MDT0001 complete [10546.925459] Lustre: server umount lustre-OST0000 complete [10564.083705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10564.666834] LustreError: 345684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10564.780623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10569.453953] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10580.110790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10585.587761] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10591.696172] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10596.731302] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10604.325811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10604.777073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10604.786102] Lustre: Skipped 1 previous similar message [10609.913150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10611.698705] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10617.738411] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10625.504794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10625.520337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10625.531494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10625.549853] Lustre: Skipped 1 previous similar message [10625.569939] Lustre: Skipped 3 previous similar messages [10630.635095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10630.649665] Lustre: Skipped 1 previous similar message [10635.745332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10635.759419] Lustre: Skipped 1 previous similar message [10639.327391] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10639.514959] Lustre: server umount lustre-OST0000 complete [10645.984687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10645.986620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10646.015323] Lustre: Skipped 1 previous similar message [10650.020478] Lustre: server umount lustre-MDT0000 complete [10651.104559] LustreError: 345685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10651.137862] LustreError: 345685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10654.545186] LustreError: 345665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120883 with bad export cookie 16962328938799692803 [10654.560630] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10654.867469] Lustre: server umount lustre-MDT0001 complete [10669.039684] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [10672.007625] Key type lgssc unregistered [10672.275251] LNet: 348874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10672.287564] LNetError: 348874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10673.319395] LNet: Removed LNI 192.168.204.144@tcp [10674.388973] Key type .llcrypt unregistered [10674.391710] Key type ._llcrypt unregistered [10706.182331] Key type ._llcrypt registered [10706.185175] Key type .llcrypt registered [10706.293743] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10707.705396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10707.801380] alg: No test for adler32 (adler32-zlib) [10708.966097] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [10709.311968] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [10711.039565] Key type lgssc registered [10712.408133] Lustre: Echo OBD driver; http://www.lustre.org/ [10715.745904] alg: No test for zstdfast (zstdfast-lustre-generic) [10715.759260] alg: No test for zstdfast (zstdfast-lustre-scomp) [10722.146207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10722.179422] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10722.195476] LustreError: 350381:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10722.219768] LustreError: 350381:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10722.237449] LustreError: 350381:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10722.251510] LustreError: 350381:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10722.263555] LustreError: 350381:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10730.886784] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10730.896858] LustreError: 350778:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10730.919335] LustreError: 350778:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10730.934844] LustreError: 350778:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10730.946233] LustreError: 350778:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10730.958738] LustreError: 350778:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10752.058101] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [10755.197572] Key type lgssc unregistered [10755.566846] LNet: 351764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10755.578397] LNetError: 351764:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10755.602423] LNet: Removed LNI 192.168.204.144@tcp [10756.525282] Key type .llcrypt unregistered [10756.527828] Key type ._llcrypt unregistered [10776.331378] Key type ._llcrypt registered [10776.340218] Key type .llcrypt registered [10776.466640] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [10794.278735] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10795.707135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10795.732635] alg: No test for adler32 (adler32-zlib) [10796.938570] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [10797.198391] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [10798.992444] Key type lgssc registered [10800.610172] Lustre: Echo OBD driver; http://www.lustre.org/ [10803.959835] alg: No test for zstdfast (zstdfast-lustre-generic) [10803.986492] alg: No test for zstdfast (zstdfast-lustre-scomp) [10840.464915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10840.499985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10841.923288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10841.963551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10842.094947] Lustre: lustre-MDT0000: new disk, initializing [10842.218882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10842.241549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10847.418353] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10859.169387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10859.291396] Lustre: 354971:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10859.334963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10859.339309] Lustre: Skipped 1 previous similar message [10859.415218] Lustre: lustre-MDT0001: new disk, initializing [10859.500966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10859.533800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10859.549161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10865.201966] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10873.008543] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10876.483754] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10883.973640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10884.532529] Lustre: lustre-OST0000: new disk, initializing [10884.554559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10884.571956] Lustre: 356714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10884.787977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10892.366925] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10892.893661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10892.909089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10893.043897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10901.917808] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10907.171865] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10907.423316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10912.506770] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10912.743613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10918.889738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10918.891606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.899827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10918.904585] LustreError: Skipped 1 previous similar message [10918.944710] Lustre: Skipped 1 previous similar message [10921.291983] Lustre: server umount lustre-OST0000 complete [10926.050868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10926.061642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10926.073742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10926.085736] Lustre: Skipped 1 previous similar message [10927.080017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10927.082287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10931.812059] Lustre: server umount lustre-MDT0000 complete [10935.927185] LustreError: 354962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121164 with bad export cookie 740688012763341844 [10935.929240] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10935.939737] LustreError: 354962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10936.273771] Lustre: server umount lustre-MDT0001 complete [10945.908960] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10948.000875] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:06:14 (1789121174) [10954.774164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.224272] LustreError: 359030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10955.244641] LustreError: 359030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10955.341533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10960.363520] LustreError: 359029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10960.832946] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10965.485160] LustreError: 359030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10970.593385] LustreError: 359029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10970.678058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10971.017544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10975.701557] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10981.681759] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10985.585679] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10993.067723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10993.321354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11000.248924] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11005.838806] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11012.838348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11013.006133] Lustre: lustre-OST0001: new disk, initializing [11013.011384] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11013.018718] Lustre: 361524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11013.111271] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11018.901779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11018.925249] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11019.067134] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11019.416145] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11027.409979] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11034.601940] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11034.609646] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11034.627423] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11034.629673] Lustre: Skipped 1 previous similar message [11034.648254] Lustre: Skipped 4 previous similar messages [11038.235758] Lustre: server umount lustre-OST0001 complete [11039.716868] LustreError: 361799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.743670] LustreError: 361799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11049.953947] LustreError: 360625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11049.973206] LustreError: 360625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11057.120824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11057.132605] LustreError: Skipped 1 previous similar message [11057.143212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11057.147941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11057.147949] Lustre: Skipped 1 previous similar message [11057.184606] Lustre: Skipped 1 previous similar message [11062.067504] Lustre: server umount lustre-OST0000 complete [11068.384713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11068.393955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11068.420519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11068.428670] Lustre: Skipped 3 previous similar messages [11072.058430] Lustre: server umount lustre-MDT0000 complete [11075.554456] LustreError: 359024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.577432] LustreError: 359024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11075.894264] LustreError: 361535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121304 with bad export cookie 740688012763342726 [11075.905429] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11075.921891] LustreError: 361535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11090.399162] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11090.770591] Lustre: server umount lustre-MDT0001 complete [11103.251887] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11107.172606] Key type lgssc unregistered [11107.646534] LNet: 363388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11107.663659] LNetError: 363388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11107.711652] LNet: Removed LNI 192.168.204.144@tcp [11108.885522] Key type .llcrypt unregistered [11108.888771] Key type ._llcrypt unregistered [11165.396163] Key type ._llcrypt registered [11165.404523] Key type .llcrypt registered [11165.515695] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [11166.546822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11166.660323] alg: No test for adler32 (adler32-zlib) [11167.826654] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [11168.180439] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [11170.055664] Key type lgssc registered [11171.333130] Lustre: Echo OBD driver; http://www.lustre.org/ [11174.905538] alg: No test for zstdfast (zstdfast-lustre-generic) [11174.931544] alg: No test for zstdfast (zstdfast-lustre-scomp) [11180.916264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11180.933646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11182.127129] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11182.174926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11182.348609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11186.045585] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11195.176562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11195.330301] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11195.377922] Lustre: 366255:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11195.742482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11201.266216] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11207.032621] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11211.285852] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11214.830671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11214.841237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11214.862858] Lustre: Skipped 1 previous similar message [11214.887234] Lustre: Skipped 1 previous similar message [11219.937500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11219.952412] Lustre: Skipped 1 previous similar message [11225.056334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11228.127825] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11228.409964] Lustre: server umount lustre-MDT0000 complete [11232.110733] LustreError: 366247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121460 with bad export cookie 9657325952244678437 [11232.113185] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11232.124429] LustreError: 366247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11232.577097] Lustre: server umount lustre-MDT0001 complete [11242.830233] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:11:09 (1789121469) [11249.815252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11250.346223] LustreError: 368542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11250.388074] LustreError: 368542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11250.459076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11255.501714] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11255.796099] LustreError: 368543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11260.914985] LustreError: 368542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11264.930191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11265.285662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11270.144673] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11275.480906] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11278.668943] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11285.747218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11285.897031] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11286.069460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11291.196422] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11298.406434] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11306.887480] Lustre: Permanently deactivating lustre-OST0000 [11306.923115] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11312.674195] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11313.595719] LustreError: 371142:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11313.601947] LustreError: 371142:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11313.605449] LustreError: 371142:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11316.276994] LustreError: 371191:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11316.286291] LustreError: 371191:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11324.333292] Lustre: server umount lustre-OST0000 complete [11326.699744] LustreError: 371440:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [11326.706182] LustreError: 371440:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11331.042161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11331.052167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11331.055628] Lustre: Skipped 1 previous similar message [11331.068610] Lustre: Skipped 2 previous similar messages [11336.169584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11336.184306] Lustre: Skipped 1 previous similar message [11343.329353] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11343.501749] Lustre: server umount lustre-MDT0000 complete [11346.400105] LustreError: 368538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11346.424266] LustreError: 368538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11347.187294] LustreError: 368524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121575 with bad export cookie 9657325952244679018 [11347.188476] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11347.196684] LustreError: 368524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11357.879795] Lustre: server umount lustre-MDT0001 complete [11367.919682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11372.749289] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11376.258804] LustreError: 372584:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11382.602577] Lustre: 372923:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.144@tcp [11384.903962] Lustre: server umount MGS complete [11396.896320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11397.667564] LustreError: 373648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11397.711711] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11397.723107] Lustre: Skipped 1 previous similar message [11397.809427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11402.961476] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11408.354644] LustreError: 373648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11408.374717] LustreError: 373648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11413.511629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11413.879842] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11413.996264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11419.111459] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11425.379990] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11429.897474] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11432.622499] Lustre: Permanently reactivating lustre-OST0000 [11432.643295] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11432.651578] Lustre: Skipped 2 previous similar messages [11461.096513] LustreError: 375202:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11468.833595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11469.348418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11476.387875] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11483.612183] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11490.484429] Lustre: server umount lustre-OST0000 complete [11495.907438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11495.916116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11495.916830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11495.916835] Lustre: Skipped 2 previous similar messages [11495.917299] Lustre: Skipped 1 previous similar message [11505.255850] Lustre: server umount lustre-MDT0000 complete [11506.144405] LustreError: 375055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.176113] LustreError: 375055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11506.259057] Lustre: 365528:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789121719/real 1789121719] req@00000000a3c56934 x1876029757027968/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789121735 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11506.297164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11508.582609] LustreError: 373629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121737 with bad export cookie 9657325952244680068 [11508.594453] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11508.596678] LustreError: 373629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11509.014445] Lustre: server umount lustre-MDT0001 complete [11520.552380] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11523.174436] Key type lgssc unregistered [11523.461933] LNet: 377705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11523.469926] LNetError: 377705:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11523.487726] LNet: Removed LNI 192.168.204.144@tcp [11524.245840] Key type .llcrypt unregistered [11524.249954] Key type ._llcrypt unregistered [11542.393939] Key type ._llcrypt registered [11542.395509] Key type .llcrypt registered [11542.541499] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [11560.368668] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [11561.745743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11561.819124] alg: No test for adler32 (adler32-zlib) [11563.041180] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [11563.318989] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [11564.959222] Key type lgssc registered [11566.074093] Lustre: Echo OBD driver; http://www.lustre.org/ [11569.688762] alg: No test for zstdfast (zstdfast-lustre-generic) [11569.701221] alg: No test for zstdfast (zstdfast-lustre-scomp) [11608.068699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11608.118819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11609.679917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11609.749517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11609.885452] Lustre: lustre-MDT0000: new disk, initializing [11609.982173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11610.019598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11614.876632] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11625.048203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11625.245435] Lustre: 380912:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11625.336939] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11625.347318] Lustre: Skipped 1 previous similar message [11625.485821] Lustre: lustre-MDT0001: new disk, initializing [11625.650100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11625.736078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11625.762501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11630.611990] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11639.574726] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11643.353596] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11651.203958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11651.604420] Lustre: lustre-OST0000: new disk, initializing [11651.613583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11651.622452] Lustre: 382651:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11651.759702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11658.688083] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11660.873339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11660.889452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11660.975439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11666.281756] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11670.851306] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11671.061498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11675.003125] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11675.173223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11676.641968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11676.642335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11676.646931] LustreError: Skipped 1 previous similar message [11676.653373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11681.774154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11681.784652] Lustre: Skipped 2 previous similar messages [11682.848865] Lustre: server umount lustre-OST0000 complete [11686.881294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11686.893179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11686.901492] Lustre: Skipped 2 previous similar messages [11686.923615] Lustre: Skipped 1 previous similar message [11692.004620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11692.015264] Lustre: Skipped 1 previous similar message [11692.703378] Lustre: server umount lustre-MDT0000 complete [11696.514958] LustreError: 380903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121925 with bad export cookie 9449116506438047311 [11696.520675] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11696.526770] LustreError: 380903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11696.930609] Lustre: server umount lustre-MDT0001 complete [11706.016373] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11708.020543] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:18:54 (1789121934) [11715.568199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11716.037052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11720.576736] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11721.184549] LustreError: 384966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11721.220069] LustreError: 384966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11725.281156] LustreError: 384965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11729.966644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11730.293316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11735.018069] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11740.673469] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11744.775153] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11752.213425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11752.729981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11759.183930] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11764.619905] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11766.519734] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11769.182608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11776.494159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11776.495183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11776.498128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11776.508107] LustreError: Skipped 1 previous similar message [11776.538067] Lustre: Skipped 1 previous similar message [11780.985480] Lustre: server umount lustre-OST0000 complete [11786.722179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.739592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11786.750390] Lustre: Skipped 2 previous similar messages [11791.303860] Lustre: server umount lustre-MDT0000 complete [11791.844231] LustreError: 386242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11791.873595] LustreError: 386242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11795.761362] LustreError: 384946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122024 with bad export cookie 9449116506438048193 [11795.764035] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11795.767122] LustreError: 384946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11796.006655] Lustre: server umount lustre-MDT0001 complete [11808.213920] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11810.663642] Key type lgssc unregistered [11810.999972] LNet: 388335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11811.019371] LNetError: 388335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11812.069856] LNet: Removed LNI 192.168.204.144@tcp [11813.003918] Key type .llcrypt unregistered [11813.006977] Key type ._llcrypt unregistered [11839.224219] Key type ._llcrypt registered [11839.225616] Key type .llcrypt registered [11839.413865] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11839.818332] Key type .llcrypt unregistered [11839.825737] Key type ._llcrypt unregistered [11861.525139] Key type ._llcrypt registered [11861.526128] Key type .llcrypt registered [11861.631906] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [11862.900858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11862.949334] alg: No test for adler32 (adler32-zlib) [11864.146879] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [11864.406199] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [11866.088631] Key type lgssc registered [11867.343385] Lustre: Echo OBD driver; http://www.lustre.org/ [11870.364397] alg: No test for zstdfast (zstdfast-lustre-generic) [11870.390708] alg: No test for zstdfast (zstdfast-lustre-scomp) [11876.467979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11876.492728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11878.103207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11882.645432] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11887.586460] LustreError: 390609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11887.603328] LustreError: 390609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11889.667392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11889.915421] LustreError: 390610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11889.955560] LustreError: 390610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11889.956697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11895.276526] LustreError: 390609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.295039] LustreError: 390609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11896.417542] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11900.391120] LustreError: 390610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11900.427251] LustreError: 390610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11902.453853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11903.816239] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11907.555709] LustreError: 390609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11907.567080] LustreError: 390609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11912.574308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11913.080173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11917.933124] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11928.033615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11928.053078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11928.066849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11928.548643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11929.161389] Lustre: server umount lustre-OST0000 complete [11933.172698] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11933.189772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11933.211728] Lustre: Skipped 1 previous similar message [11933.225365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11933.233911] Lustre: Skipped 1 previous similar message [11933.388106] Lustre: server umount lustre-MDT0000 complete [11933.665901] LustreError: 390606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11933.680126] LustreError: 390606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11937.299333] LustreError: 390591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122166 with bad export cookie 13973245565107562157 [11937.307184] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11937.318907] LustreError: 390591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11937.843979] Lustre: server umount lustre-MDT0001 complete [11949.647414] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11952.092297] Key type lgssc unregistered [11952.283735] LNet: 393591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11952.291723] LNetError: 393591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11952.307380] LNet: Removed LNI 192.168.204.144@tcp [11952.903349] Key type .llcrypt unregistered [11952.906188] Key type ._llcrypt unregistered [11981.162961] Key type ._llcrypt registered [11981.164931] Key type .llcrypt registered [11981.307422] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [11982.345956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11982.379055] alg: No test for adler32 (adler32-zlib) [11983.415541] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [11983.715599] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [11985.551193] Key type lgssc registered [11986.715223] Lustre: Echo OBD driver; http://www.lustre.org/ [11989.126718] alg: No test for zstdfast (zstdfast-lustre-generic) [11989.143952] alg: No test for zstdfast (zstdfast-lustre-scomp) [11994.740416] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11994.763769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12010.400230] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12015.584674] LustreError: 395051:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [12042.207751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12051.495725] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12058.990397] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12067.033186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12067.545890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12067.642536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12072.609920] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12072.932635] LustreError: 395077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.953361] LustreError: 395077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12078.048927] LustreError: 395076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12083.171062] LustreError: 395974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12083.200951] LustreError: 395974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12083.242232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12083.754271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12089.092572] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12095.588522] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12100.202540] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12108.412724] Lustre: server umount lustre-OST0000 complete [12113.899276] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12113.904884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12113.922218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12119.520593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12119.524441] Lustre: Skipped 3 previous similar messages [12122.717357] Lustre: server umount lustre-MDT0000 complete [12124.641072] LustreError: 395976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12124.655340] LustreError: 395976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12125.600289] Lustre: 394339:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789122338/real 1789122338] req@000000005b20e0e3 x1876030612599808/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789122354 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12125.647380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12125.665761] Lustre: Skipped 1 previous similar message [12126.572884] LustreError: 395058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122355 with bad export cookie 8240661211102793122 [12126.584241] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12126.585453] LustreError: 395058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12127.011529] Lustre: server umount lustre-MDT0001 complete [12141.350931] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12144.932876] Key type lgssc unregistered [12145.312723] LNet: 398266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12145.323345] LNetError: 398266:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12145.353070] LNet: Removed LNI 192.168.204.144@tcp [12146.287589] Key type .llcrypt unregistered [12146.289982] Key type ._llcrypt unregistered [12172.357494] Key type ._llcrypt registered [12172.363722] Key type .llcrypt registered [12172.532339] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12174.242174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12174.273048] alg: No test for adler32 (adler32-zlib) [12175.264730] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [12175.481638] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [12177.223208] Key type lgssc registered [12178.897893] Lustre: Echo OBD driver; http://www.lustre.org/ [12182.787895] alg: No test for zstdfast (zstdfast-lustre-generic) [12182.835784] alg: No test for zstdfast (zstdfast-lustre-scomp) [12189.706671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12189.727734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12191.361746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12196.893628] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12201.954708] LustreError: 399752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12201.965712] LustreError: 399752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12207.075896] LustreError: 399753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12207.955995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12208.993809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12216.087900] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12223.715707] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12228.928502] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12237.116190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12237.575888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12244.590778] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12250.523232] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12262.149911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12267.495412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12267.521664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12267.531985] Lustre: Skipped 2 previous similar messages [12267.555531] Lustre: Skipped 3 previous similar messages [12271.899389] Lustre: server umount lustre-MDT0000 complete [12272.605928] LustreError: 401369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12272.642174] LustreError: 401369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12274.559315] LustreError: 399733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.44@tcp arrived at 1789122503 with bad export cookie 13707627360416565235 [12274.588566] LustreError: 399733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12276.370595] LustreError: 399734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122505 with bad export cookie 13707627360416564577 [12276.384754] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12276.389592] LustreError: 399734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12277.728622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12277.733870] LustreError: 399749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12277.746056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12277.761146] LustreError: 399749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12281.765737] LustreError: 401369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12281.766228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12281.784597] LustreError: 401369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12282.643589] Lustre: server umount lustre-OST0000 complete [12288.682470] Lustre: server umount lustre-MDT0001 complete [12299.252222] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12302.572544] Key type lgssc unregistered [12302.924770] LNet: 403104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12302.938125] LNetError: 403104:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12302.957676] LNet: Removed LNI 192.168.204.144@tcp [12304.444860] Key type .llcrypt unregistered [12304.447124] Key type ._llcrypt unregistered [12336.500938] Key type ._llcrypt registered [12336.504180] Key type .llcrypt registered [12336.616604] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12337.484333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12337.555528] alg: No test for adler32 (adler32-zlib) [12338.569666] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [12338.818274] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [12340.591178] Key type lgssc registered [12342.082385] Lustre: Echo OBD driver; http://www.lustre.org/ [12345.950870] alg: No test for zstdfast (zstdfast-lustre-generic) [12345.960517] alg: No test for zstdfast (zstdfast-lustre-scomp) [12352.153575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12352.187263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12353.782974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12358.295101] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12363.240951] LustreError: 404605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12363.269389] LustreError: 404605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12367.416766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12368.160643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12372.166787] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12377.513673] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12381.693816] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12389.530654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12389.958930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12397.135915] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12399.222164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12403.269870] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12409.315324] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12409.333276] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12409.354183] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12409.836115] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12413.638379] Lustre: server umount lustre-MDT0001 complete [12414.959462] LustreError: 404602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12415.001184] LustreError: 404602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12420.072803] LustreError: 404605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12420.102045] LustreError: 404605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12425.185861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12425.198815] LustreError: 404606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12425.202871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12425.229068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12425.240544] Lustre: Skipped 2 previous similar messages [12427.056317] Lustre: server umount lustre-OST0000 complete [12430.308117] LustreError: 404605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12431.216633] Lustre: server umount lustre-MDT0000 complete [12443.286605] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12446.564514] Key type lgssc unregistered [12446.739720] LNet: 407932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12446.744869] LNetError: 407932:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12446.759915] LNet: Removed LNI 192.168.204.144@tcp [12447.895863] Key type .llcrypt unregistered [12447.898486] Key type ._llcrypt unregistered [12477.838500] Key type ._llcrypt registered [12477.841351] Key type .llcrypt registered [12477.941516] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [12492.529903] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12493.613632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12493.657250] alg: No test for adler32 (adler32-zlib) [12494.744837] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [12494.988936] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [12496.679399] Key type lgssc registered [12497.663672] Lustre: Echo OBD driver; http://www.lustre.org/ [12499.885705] alg: No test for zstdfast (zstdfast-lustre-generic) [12499.898989] alg: No test for zstdfast (zstdfast-lustre-scomp) [12536.400192] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12554.277899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12554.300668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12555.599621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12555.673066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12555.827591] Lustre: lustre-MDT0000: new disk, initializing [12555.922075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12555.970733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12560.087748] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12576.598036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12576.906217] Lustre: lustre-OST0000: new disk, initializing [12576.914475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12576.924130] Lustre: Skipped 1 previous similar message [12576.947665] Lustre: 412996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12577.076233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12584.657339] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12585.678423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12585.693509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12585.734936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12593.534740] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12607.919262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12608.122261] Lustre: 411995:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12608.238157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12608.319390] Lustre: lustre-MDT0001: new disk, initializing [12608.517757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12608.600226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12608.615654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12614.403818] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12627.218542] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12627.488428] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12633.784773] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12634.123144] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12641.299665] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [12641.763050] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12641.768806] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12641.791878] Lustre: Skipped 1 previous similar message [12641.802448] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12641.809203] Lustre: Skipped 1 previous similar message [12645.879402] Lustre: server umount lustre-MDT0001 complete [12652.003460] LustreError: 412003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.017084] LustreError: 412003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12657.125452] LustreError: 414517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12657.137874] LustreError: 414517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12662.244032] LustreError: 412002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12662.285676] LustreError: 412002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12667.364910] LustreError: 412007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12667.368750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12667.418830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12669.442449] Lustre: server umount lustre-OST0000 complete [12672.481357] LustreError: 415405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12679.628302] Lustre: server umount lustre-MDT0000 complete [12695.794831] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12699.434774] Key type lgssc unregistered [12699.748284] LNet: 416566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12699.752348] LNetError: 416566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12699.763867] LNet: Removed LNI 192.168.204.144@tcp [12700.850422] Key type .llcrypt unregistered [12700.858412] Key type ._llcrypt unregistered [12743.500305] Key type ._llcrypt registered [12743.505639] Key type .llcrypt registered [12743.611314] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12743.845544] Key type .llcrypt unregistered [12743.849783] Key type ._llcrypt unregistered [12762.991436] Key type ._llcrypt registered [12762.993331] Key type .llcrypt registered [12763.070940] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12764.823466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12764.978443] alg: No test for adler32 (adler32-zlib) [12766.260353] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [12766.651782] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [12768.464698] Key type lgssc registered [12771.134820] Lustre: Echo OBD driver; http://www.lustre.org/ [12776.783677] alg: No test for zstdfast (zstdfast-lustre-generic) [12776.804873] alg: No test for zstdfast (zstdfast-lustre-scomp) [12807.354327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12807.381503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12808.743905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12808.837630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12808.916429] Lustre: lustre-MDT0000: new disk, initializing [12808.999421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12809.020526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12814.229761] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12826.698646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12826.849244] Lustre: 420005:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12826.905408] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12826.909798] Lustre: Skipped 1 previous similar message [12826.984898] Lustre: lustre-MDT0001: new disk, initializing [12827.087369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12827.179056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12827.197748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12832.424868] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12841.277629] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12845.718481] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12853.123364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12853.484363] Lustre: lustre-OST0000: new disk, initializing [12853.489147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12853.500378] Lustre: 421747:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12853.628983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12859.538536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12859.560267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12859.730571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12860.958520] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12869.475740] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12878.306293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12878.319179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12878.337809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12880.362266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12880.367066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12880.388573] Lustre: Skipped 1 previous similar message [12880.416128] Lustre: Skipped 2 previous similar messages [12882.880781] Lustre: server umount lustre-MDT0000 complete [12887.523701] LustreError: 419998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123116 with bad export cookie 4197023788857811008 [12887.540138] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12887.985945] Lustre: server umount lustre-MDT0001 complete [12901.638663] Lustre: server umount lustre-OST0000 complete [12913.207481] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:39:00 (1789123140) [12921.729049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12922.375362] LustreError: 423662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12922.406074] LustreError: 423662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12922.467670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12927.456701] LustreError: 423663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12927.735626] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12932.579601] LustreError: 423662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12937.699799] LustreError: 423663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12938.402080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12938.992953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12944.099934] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12949.735704] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12953.275917] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12973.800074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12974.209806] Lustre: Found index 0 for lustre-OST0000, updating log [12974.470149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12981.743869] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12988.725561] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12993.968596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12999.147105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12999.149700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12999.186369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12999.207664] Lustre: Skipped 1 previous similar message [13001.356613] Lustre: server umount lustre-OST0000 complete [13017.453889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13017.468381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13017.477587] Lustre: Skipped 2 previous similar messages [13017.511473] Lustre: Skipped 3 previous similar messages [13022.690034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13022.700716] Lustre: Skipped 1 previous similar message [13023.865691] Lustre: server umount lustre-MDT0000 complete [13026.787288] LustreError: 423659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13026.830955] LustreError: 423659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13028.393089] LustreError: 423643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123257 with bad export cookie 4197023788857812401 [13028.393481] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13028.404776] LustreError: 423643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13028.671289] Lustre: server umount lustre-MDT0001 complete [13040.308676] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:41:06 (1789123266) [13042.207159] LNet: 1 local NIs in recovery (showing 1): 192.168.204.144@tcp [13665.735483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13666.272727] LustreError: 429736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13666.285907] LustreError: 429736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13666.355243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13671.312965] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [13671.398036] LustreError: 429737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13676.518590] LustreError: 429736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13681.730585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13682.166239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13687.356256] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [13693.136729] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13697.095498] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13717.168522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13717.459879] Lustre: Found index 0 for lustre-OST0000, updating log [13717.901758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13720.099734] LNet: 1 local NIs in recovery (showing 1): 192.168.204.144@tcp [13724.870663] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [13730.814226] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13733.244577] Lustre: server umount lustre-OST0000 complete [13738.084346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13738.087639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13738.100411] Lustre: Skipped 1 previous similar message [13738.125294] Lustre: Skipped 1 previous similar message [13739.169180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13739.181659] Lustre: Skipped 1 previous similar message [13741.223295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13741.228056] Lustre: Skipped 3 previous similar messages [13743.403492] LustreError: 429732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13743.429818] LustreError: 429732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13743.478205] Lustre: server umount lustre-MDT0000 complete [13748.376110] LustreError: 430455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123977 with bad export cookie 4197023788857813486 [13748.382480] LustreError: 430455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13748.385199] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13749.036360] Lustre: server umount lustre-MDT0001 complete [13779.553532] LNet: 1 local NIs in recovery (showing 1): 192.168.204.144@tcp [13781.119691] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [13792.151834] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [13832.364452] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:54:18 (1789124058) [13851.960030] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [13854.874952] Key type lgssc unregistered [13855.198716] LNet: 435978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13855.208653] LNetError: 435978:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13981.162641] LNet: Removed LNI 192.168.204.144@tcp [13982.152986] Key type .llcrypt unregistered [13982.154926] Key type ._llcrypt unregistered [13999.645369] Key type ._llcrypt registered [13999.648869] Key type .llcrypt registered [13999.781785] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14000.679965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14000.760231] alg: No test for adler32 (adler32-zlib) [14001.852148] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [14002.102115] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [14003.815211] Key type lgssc registered [14004.975129] Lustre: Echo OBD driver; http://www.lustre.org/ [14008.051341] alg: No test for zstdfast (zstdfast-lustre-generic) [14008.071582] alg: No test for zstdfast (zstdfast-lustre-scomp) [14010.894829] LNet: Added LNI 192.168.204.144@tcp100 [8/256/0/180] [14012.075175] LNet: Added LNI 192.168.204.144@tcp101 [8/256/0/180] [14013.102433] LNet: Added LNI 192.168.204.144@tcp102 [8/256/0/180] [14014.123755] LNet: Added LNI 192.168.204.144@tcp103 [8/256/0/180] [14014.981773] LNet: Added LNI 192.168.204.144@tcp104 [8/256/0/180] [14016.079489] LNet: Added LNI 192.168.204.144@tcp105 [8/256/0/180] [14016.927428] LNet: Added LNI 192.168.204.144@tcp106 [8/256/0/180] [14017.826036] LNet: Added LNI 192.168.204.144@tcp107 [8/256/0/180] [14019.062302] LNet: Added LNI 192.168.204.144@tcp108 [8/256/0/180] [14019.835186] LNet: Added LNI 192.168.204.144@tcp109 [8/256/0/180] [14021.097170] LNet: Added LNI 192.168.204.144@tcp110 [8/256/0/180] [14022.076245] LNet: Added LNI 192.168.204.144@tcp111 [8/256/0/180] [14022.953757] LNet: Added LNI 192.168.204.144@tcp112 [8/256/0/180] [14023.791190] LNet: Added LNI 192.168.204.144@tcp113 [8/256/0/180] [14024.906082] LNet: Added LNI 192.168.204.144@tcp114 [8/256/0/180] [14025.706213] LNet: Added LNI 192.168.204.144@tcp115 [8/256/0/180] [14026.580194] LNet: Added LNI 192.168.204.144@tcp116 [8/256/0/180] [14027.698181] LNet: Added LNI 192.168.204.144@tcp117 [8/256/0/180] [14028.579462] LNet: Added LNI 192.168.204.144@tcp118 [8/256/0/180] [14029.461755] LNet: Added LNI 192.168.204.144@tcp119 [8/256/0/180] [14030.458616] LNet: Added LNI 192.168.204.144@tcp120 [8/256/0/180] [14031.310811] LNet: Added LNI 192.168.204.144@tcp121 [8/256/0/180] [14032.179358] LNet: Added LNI 192.168.204.144@tcp122 [8/256/0/180] [14033.245639] LNet: Added LNI 192.168.204.144@tcp123 [8/256/0/180] [14034.167960] LNet: Added LNI 192.168.204.144@tcp124 [8/256/0/180] [14035.096897] LNet: Added LNI 192.168.204.144@tcp125 [8/256/0/180] [14036.110663] LNet: Added LNI 192.168.204.144@tcp126 [8/256/0/180] [14037.069640] LNet: Added LNI 192.168.204.144@tcp127 [8/256/0/180] [14038.024728] LNet: Added LNI 192.168.204.144@tcp128 [8/256/0/180] [14038.853953] LNet: Added LNI 192.168.204.144@tcp129 [8/256/0/180] [14039.659419] LNet: Added LNI 192.168.204.144@tcp130 [8/256/0/180] [14040.265561] LNet: Added LNI 192.168.204.144@tcp131 [8/256/0/180] [14041.239613] LNet: Added LNI 192.168.204.144@tcp132 [8/256/0/180] [14042.279962] LNet: Added LNI 192.168.204.144@tcp133 [8/256/0/180] [14043.196532] LNet: Added LNI 192.168.204.144@tcp134 [8/256/0/180] [14044.018950] LNet: Added LNI 192.168.204.144@tcp135 [8/256/0/180] [14051.189251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14051.240566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14052.353304] Lustre: 439081:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [14052.478231] LNetError: 436485:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [14052.500262] LNetError: Protocol error connecting to 192.168.204.144@tcp121 at host 192.168.204.144:988: is it running a compatible version of Lustre? [14052.561782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14052.584240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14052.692445] Lustre: lustre-MDT0000: new disk, initializing [14052.831465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14052.865223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14053.345518] LNet: 6 local NIs in recovery (showing 6): 192.168.204.144@tcp121, 192.168.204.144@tcp125, 192.168.204.144@tcp126, 192.168.204.144@tcp128, 192.168.204.144@tcp127, 192.168.204.144@tcp129 [14053.348845] LNetError: 439108:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [14053.386113] LNetError: 439108:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14053.406713] LNetError: Protocol error connecting to 192.168.204.144@tcp107 at host 192.168.204.144:988: is it running a compatible version of Lustre? [14053.427245] LNetError: Skipped 8 previous similar messages [14054.433106] LNetError: 436488:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [14054.449055] LNetError: 436488:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14054.457814] LNetError: Protocol error connecting to 192.168.204.144@tcp128 at host 192.168.204.144:988: is it running a compatible version of Lustre? [14057.205956] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14067.769229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14067.940135] Lustre: 439099:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14068.029222] Lustre: 439884:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [14068.036160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14068.039288] Lustre: 439884:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14068.056702] Lustre: Skipped 1 previous similar message [14068.061084] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14068.128899] Lustre: lustre-MDT0001: new disk, initializing [14068.241330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14068.278612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14068.296336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14072.621843] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14080.046049] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14083.947682] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14086.053560] LNetError: 439113:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [14086.063527] LNetError: Protocol error connecting to 192.168.204.144@tcp107 at host 192.168.204.144:988: is it running a compatible version of Lustre? [14090.284700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14090.598233] Lustre: lustre-OST0000: new disk, initializing [14090.605850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14090.619054] Lustre: 440845:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14090.713601] Lustre: 440850:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [14090.719878] Lustre: 440850:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [14090.767452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14093.218091] LNetError: 436486:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [14093.240669] LNetError: 436486:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [14093.251381] LNetError: Protocol error connecting to 192.168.204.144@tcp124 at host 192.168.204.144:988: is it running a compatible version of Lustre? [14093.259441] LNetError: Skipped 2 previous similar messages [14096.546615] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14099.971290] Lustre: 441314:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [14099.976562] Lustre: 441314:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [14100.027109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14100.043724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14100.210063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14101.421410] LNetError: 436487:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [14101.438269] LNetError: 436487:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [14101.459933] LNetError: Protocol error connecting to 192.168.204.144@tcp128 at host 192.168.204.144:988: is it running a compatible version of Lustre? [14101.477523] LNetError: Skipped 3 previous similar messages [14104.054750] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14115.810872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14115.817113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14115.855508] Lustre: Skipped 1 previous similar message [14115.870479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14117.472165] Lustre: server umount lustre-OST0000 complete [14118.819447] LNetError: 439114:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [14118.845134] LNetError: 439114:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [14118.859352] LNetError: Protocol error connecting to 192.168.204.144@tcp124 at host 192.168.204.144:988: is it running a compatible version of Lustre? [14118.882689] LNetError: Skipped 4 previous similar messages [14124.513945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14124.525102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14124.545846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14124.552314] Lustre: Skipped 1 previous similar message [14126.049151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14128.119151] Lustre: server umount lustre-MDT0000 complete [14131.975895] LustreError: 439882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124360 with bad export cookie 14038153375393164279 [14131.992642] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14131.994821] LustreError: 439882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14132.377088] Lustre: server umount lustre-MDT0001 complete [14142.970797] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [14145.667668] Key type lgssc unregistered [14146.032912] LNet: 442627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14146.039364] LNetError: 442627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14146.044574] LNet: Removed LNI 192.168.204.144@tcp [14147.103215] LNet: Removed LNI 192.168.204.144@tcp100 [14147.106506] LNetError: 442627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14147.112343] LNetError: 442627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14147.119416] LNet: Removed LNI 192.168.204.144@tcp101 [14148.127138] LNet: Removed LNI 192.168.204.144@tcp102 [14148.130813] LNetError: 442627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14148.144404] LNetError: 442627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14148.154184] LNet: Removed LNI 192.168.204.144@tcp103 [14149.215158] LNet: Removed LNI 192.168.204.144@tcp104 [14150.239241] LNet: Removed LNI 192.168.204.144@tcp105 [14150.244820] LNetError: 442627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14150.253207] LNetError: 442627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14150.260476] LNet: Removed LNI 192.168.204.144@tcp106 [14151.332533] LNet: Removed LNI 192.168.204.144@tcp107 [14152.351259] LNet: Removed LNI 192.168.204.144@tcp108 [14153.375398] LNet: Removed LNI 192.168.204.144@tcp109 [14154.399255] LNet: Removed LNI 192.168.204.144@tcp110 [14154.416462] LNetError: 442627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14154.430143] LNetError: 442627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14154.444393] LNet: Removed LNI 192.168.204.144@tcp111 [14155.488971] LNet: Removed LNI 192.168.204.144@tcp112 [14156.511277] LNet: Removed LNI 192.168.204.144@tcp113 [14157.535861] LNet: Removed LNI 192.168.204.144@tcp114 [14158.560826] LNet: Removed LNI 192.168.204.144@tcp115 [14159.583152] LNet: Removed LNI 192.168.204.144@tcp116 [14160.607211] LNet: Removed LNI 192.168.204.144@tcp117 [14161.631153] LNet: Removed LNI 192.168.204.144@tcp118 [14162.657465] LNet: Removed LNI 192.168.204.144@tcp119 [14162.662230] LNetError: 442627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14162.676096] LNetError: 442627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14162.691240] LNet: Removed LNI 192.168.204.144@tcp120 [14163.744116] LNet: Removed LNI 192.168.204.144@tcp121 [14164.768543] LNet: Removed LNI 192.168.204.144@tcp122 [14165.791278] LNet: Removed LNI 192.168.204.144@tcp123 [14166.815153] LNet: Removed LNI 192.168.204.144@tcp124 [14167.839159] LNet: Removed LNI 192.168.204.144@tcp125 [14168.863292] LNet: Removed LNI 192.168.204.144@tcp126 [14169.887184] LNet: Removed LNI 192.168.204.144@tcp127 [14170.912830] LNet: Removed LNI 192.168.204.144@tcp128 [14171.935983] LNet: Removed LNI 192.168.204.144@tcp129 [14171.946383] LNet: Removed LNI 192.168.204.144@tcp130 [14171.953563] LNet: Removed LNI 192.168.204.144@tcp131 [14171.963198] LNet: Removed LNI 192.168.204.144@tcp132 [14171.966991] LNet: Removed LNI 192.168.204.144@tcp133 [14171.973551] LNet: Removed LNI 192.168.204.144@tcp134 [14171.989383] LNet: Removed LNI 192.168.204.144@tcp135 [14173.025548] Key type .llcrypt unregistered [14173.028219] Key type ._llcrypt unregistered [14198.720447] Key type ._llcrypt registered [14198.722065] Key type .llcrypt registered [14198.908212] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [14219.819393] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14221.544655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14221.634894] alg: No test for adler32 (adler32-zlib) [14223.096091] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [14223.595212] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [14225.423916] Key type lgssc registered [14227.019803] Lustre: Echo OBD driver; http://www.lustre.org/ [14230.579378] alg: No test for zstdfast (zstdfast-lustre-generic) [14230.631640] alg: No test for zstdfast (zstdfast-lustre-scomp) [14264.187333] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:01:31 (1789124491) [14284.118633] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [14287.183512] Key type lgssc unregistered [14287.543511] LNet: 446758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14287.549432] LNetError: 446758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14287.563578] LNet: Removed LNI 192.168.204.144@tcp [14288.406568] Key type .llcrypt unregistered [14288.409814] Key type ._llcrypt unregistered [14305.919547] Key type ._llcrypt registered [14305.925537] Key type .llcrypt registered [14306.082344] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14307.443531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14307.479649] alg: No test for adler32 (adler32-zlib) [14308.642491] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [14308.863647] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [14310.559319] Key type lgssc registered [14311.930786] Lustre: Echo OBD driver; http://www.lustre.org/ [14314.936801] alg: No test for zstdfast (zstdfast-lustre-generic) [14314.960851] alg: No test for zstdfast (zstdfast-lustre-scomp) [14322.137944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14322.196509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14323.498708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14323.539957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14323.613308] Lustre: lustre-MDT0000: new disk, initializing [14323.731929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14323.748673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14328.501251] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14339.103486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14339.235601] Lustre: 448041:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14339.284460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14339.289677] Lustre: Skipped 1 previous similar message [14339.361431] Lustre: lustre-MDT0001: new disk, initializing [14339.471624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14339.512494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14339.533185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14344.229196] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14351.535957] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14355.367806] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14362.690385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14363.048571] Lustre: lustre-OST0000: new disk, initializing [14363.058356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14363.068314] Lustre: 449784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14363.216856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14369.588824] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14371.386267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14371.401392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14371.514700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14377.414132] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14381.753476] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing lnet_if_list [14387.387595] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14387.419495] LNet: Added LNI 192.168.204.144@tcp10 [8/256/0/180] [14394.094160] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14394.183772] LNet: Added LNI 192.168.204.144@tcp11 [8/256/0/180] [14399.746678] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14399.773128] LNet: Added LNI 192.168.204.144@tcp12 [8/256/0/180] [14405.251892] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14405.331760] LNet: Added LNI 192.168.204.144@tcp13 [8/256/0/180] [14410.896570] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14410.925214] LNet: Added LNI 192.168.204.144@tcp14 [8/256/0/180] [14415.926438] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14415.961995] LNet: Added LNI 192.168.204.144@tcp15 [8/256/0/180] [14421.142984] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14421.186236] LNet: Added LNI 192.168.204.144@tcp16 [8/256/0/180] [14426.590352] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14426.641852] LNet: Added LNI 192.168.204.144@tcp17 [8/256/0/180] [14431.751363] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14431.786511] LNet: Added LNI 192.168.204.144@tcp18 [8/256/0/180] [14436.888988] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14436.922429] LNet: Added LNI 192.168.204.144@tcp19 [8/256/0/180] [14442.044514] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14442.080327] LNet: Added LNI 192.168.204.144@tcp20 [8/256/0/180] [14447.926757] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14447.958600] LNet: Added LNI 192.168.204.144@tcp21 [8/256/0/180] [14453.416926] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14453.460635] LNet: Added LNI 192.168.204.144@tcp22 [8/256/0/180] [14459.326233] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14459.362611] LNet: Added LNI 192.168.204.144@tcp23 [8/256/0/180] [14464.582873] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14464.604748] LNet: Added LNI 192.168.204.144@tcp24 [8/256/0/180] [14470.438557] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14470.465093] LNet: Added LNI 192.168.204.144@tcp25 [8/256/0/180] [14476.308920] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14476.359258] LNet: Added LNI 192.168.204.144@tcp26 [8/256/0/180] [14482.224957] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14482.249085] LNet: Added LNI 192.168.204.144@tcp27 [8/256/0/180] [14487.391743] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14487.422668] LNet: Added LNI 192.168.204.144@tcp28 [8/256/0/180] [14493.188384] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14493.220889] LNet: Added LNI 192.168.204.144@tcp29 [8/256/0/180] [14500.843602] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14500.932893] LNet: Added LNI 192.168.204.144@tcp30 [8/256/0/180] [14507.607675] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14507.634948] LNet: Added LNI 192.168.204.144@tcp31 [8/256/0/180] [14514.979871] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14515.052406] LNet: Added LNI 192.168.204.144@tcp32 [8/256/0/180] [14522.081233] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14522.107209] LNet: Added LNI 192.168.204.144@tcp33 [8/256/0/180] [14529.269226] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14529.301629] LNet: Added LNI 192.168.204.144@tcp34 [8/256/0/180] [14535.480166] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14535.506077] LNet: Added LNI 192.168.204.144@tcp35 [8/256/0/180] [14541.974502] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14542.027665] LNet: Added LNI 192.168.204.144@tcp36 [8/256/0/180] [14549.796876] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14549.832129] LNet: Added LNI 192.168.204.144@tcp37 [8/256/0/180] [14557.014418] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14557.051680] LNet: Added LNI 192.168.204.144@tcp38 [8/256/0/180] [14564.302846] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14564.342603] LNet: Added LNI 192.168.204.144@tcp39 [8/256/0/180] [14571.630186] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14571.652892] LNet: Added LNI 192.168.204.144@tcp40 [8/256/0/180] [14578.896265] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14578.919283] LNet: Added LNI 192.168.204.144@tcp41 [8/256/0/180] [14587.684085] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14587.719733] LNet: Added LNI 192.168.204.144@tcp42 [8/256/0/180] [14595.862665] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14595.913817] LNet: Added LNI 192.168.204.144@tcp43 [8/256/0/180] [14604.311868] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14604.350216] LNet: Added LNI 192.168.204.144@tcp44 [8/256/0/180] [14611.701674] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14611.757111] LNet: Added LNI 192.168.204.144@tcp45 [8/256/0/180] [14621.491628] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14621.538504] LNet: Added LNI 192.168.204.144@tcp46 [8/256/0/180] [14633.467947] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14633.518349] LNet: Added LNI 192.168.204.144@tcp47 [8/256/0/180] [14643.995800] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14644.038750] LNet: Added LNI 192.168.204.144@tcp48 [8/256/0/180] [14654.789583] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14654.811128] LNet: Added LNI 192.168.204.144@tcp49 [8/256/0/180] [14664.306368] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14664.331668] LNet: Added LNI 192.168.204.144@tcp50 [8/256/0/180] [14687.724965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14687.743756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14687.765819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14689.251594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14689.255501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14689.292591] Lustre: Skipped 1 previous similar message [14691.944603] Lustre: server umount lustre-MDT0000 complete [14695.707968] LustreError: 448818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124924 with bad export cookie 11934331220934340739 [14695.711310] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14695.719972] LustreError: 448818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14696.056148] Lustre: server umount lustre-MDT0001 complete [14715.871550] Lustre: 447261:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789124928/real 1789124928] req@000000001291707a x1876033050687360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789124944 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14715.913355] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14716.625321] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14716.964643] Lustre: 447261:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789124929/real 1789124929] req@000000000cbf4afe x1876033050687488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789124945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14721.056190] Lustre: 447263:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789124933/real 1789124933] req@00000000a4d7feb6 x1876033050687616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789124949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14726.751372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14727.136232] Lustre: 447260:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789124939/real 1789124939] req@00000000416c4aa1 x1876033050688000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789124955 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14727.171100] Lustre: 447260:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14731.295277] Lustre: 447260:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789124944/real 1789124944] req@0000000055075aa4 x1876033050688128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789124960 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14740.773071] LustreError: 449777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14740.793552] LustreError: 449777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14740.869786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14745.013986] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14745.016196] LustreError: 449778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14745.694893] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14750.186507] LustreError: 449777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14755.029633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14755.407528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14760.924810] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14761.473503] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14766.648525] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14770.711212] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14792.162108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14792.163515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14792.167199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14792.167208] Lustre: Skipped 2 previous similar messages [14792.205238] Lustre: Skipped 1 previous similar message [14797.291828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14797.298634] Lustre: Skipped 2 previous similar messages [14797.873940] Lustre: server umount lustre-OST0000 complete [14802.408661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14802.416601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14802.420195] Lustre: Skipped 1 previous similar message [14802.431252] Lustre: Skipped 1 previous similar message [14808.039514] Lustre: server umount lustre-MDT0000 complete [14812.167344] LustreError: 448818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125040 with bad export cookie 11934331220934343448 [14812.168506] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14812.181995] LustreError: 448818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14812.577874] Lustre: server umount lustre-MDT0001 complete [14825.886410] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [14829.622621] Key type lgssc unregistered [14830.024934] LNet: 475757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14830.039435] LNetError: 475757:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14830.049965] LNet: Removed LNI 192.168.204.144@tcp [14830.055066] LNet: Removed LNI 192.168.204.144@tcp10 [14830.060982] LNet: Removed LNI 192.168.204.144@tcp11 [14830.072690] LNet: Removed LNI 192.168.204.144@tcp12 [14830.080659] LNet: Removed LNI 192.168.204.144@tcp13 [14830.084635] LNet: Removed LNI 192.168.204.144@tcp14 [14830.088622] LNet: Removed LNI 192.168.204.144@tcp15 [14830.094415] LNet: Removed LNI 192.168.204.144@tcp16 [14830.096371] LNet: Removed LNI 192.168.204.144@tcp17 [14830.101139] LNet: Removed LNI 192.168.204.144@tcp18 [14830.112341] LNet: Removed LNI 192.168.204.144@tcp19 [14830.127644] LNet: Removed LNI 192.168.204.144@tcp20 [14830.133529] LNet: Removed LNI 192.168.204.144@tcp21 [14830.140540] LNet: Removed LNI 192.168.204.144@tcp22 [14830.149284] LNet: Removed LNI 192.168.204.144@tcp23 [14830.153436] LNet: Removed LNI 192.168.204.144@tcp24 [14830.157536] LNet: Removed LNI 192.168.204.144@tcp25 [14830.161563] LNet: Removed LNI 192.168.204.144@tcp26 [14830.165291] LNet: Removed LNI 192.168.204.144@tcp27 [14830.171164] LNet: Removed LNI 192.168.204.144@tcp28 [14830.175419] LNet: Removed LNI 192.168.204.144@tcp29 [14830.180573] LNet: Removed LNI 192.168.204.144@tcp30 [14830.185622] LNet: Removed LNI 192.168.204.144@tcp31 [14830.191679] LNet: Removed LNI 192.168.204.144@tcp32 [14830.193720] LNet: Removed LNI 192.168.204.144@tcp33 [14830.198713] LNet: Removed LNI 192.168.204.144@tcp34 [14830.205095] LNet: Removed LNI 192.168.204.144@tcp35 [14830.209576] LNet: Removed LNI 192.168.204.144@tcp36 [14830.214040] LNet: Removed LNI 192.168.204.144@tcp37 [14830.217569] LNet: Removed LNI 192.168.204.144@tcp38 [14830.222805] LNet: Removed LNI 192.168.204.144@tcp39 [14830.226442] LNet: Removed LNI 192.168.204.144@tcp40 [14830.230284] LNet: Removed LNI 192.168.204.144@tcp41 [14830.239406] LNet: Removed LNI 192.168.204.144@tcp42 [14830.246116] LNet: Removed LNI 192.168.204.144@tcp43 [14830.249422] LNet: Removed LNI 192.168.204.144@tcp44 [14830.252717] LNet: Removed LNI 192.168.204.144@tcp45 [14830.255653] LNet: Removed LNI 192.168.204.144@tcp46 [14830.258572] LNet: Removed LNI 192.168.204.144@tcp47 [14830.268129] LNet: Removed LNI 192.168.204.144@tcp48 [14830.277900] LNet: Removed LNI 192.168.204.144@tcp49 [14830.315636] LNet: Removed LNI 192.168.204.144@tcp50 [14831.126136] Key type .llcrypt unregistered [14831.129055] Key type ._llcrypt unregistered [14855.464881] Key type ._llcrypt registered [14855.468809] Key type .llcrypt registered [14855.591036] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [14872.327864] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14873.372679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14873.494662] alg: No test for adler32 (adler32-zlib) [14874.596698] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [14874.820503] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [14876.471200] Key type lgssc registered [14877.855196] Lustre: Echo OBD driver; http://www.lustre.org/ [14881.436213] alg: No test for zstdfast (zstdfast-lustre-generic) [14881.459558] alg: No test for zstdfast (zstdfast-lustre-scomp) [14915.785082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14915.841193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14917.164400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14917.223589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14917.307583] Lustre: lustre-MDT0000: new disk, initializing [14917.423565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14917.442838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14922.793166] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14934.230629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14934.391075] Lustre: 479007:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14934.449685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14934.453328] Lustre: Skipped 1 previous similar message [14934.516463] Lustre: lustre-MDT0001: new disk, initializing [14934.644241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14934.686856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14934.709964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14939.744404] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14947.502826] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14951.383843] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14959.454886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14959.713512] Lustre: *** cfs_fail_loc=90f, val=0*** [14959.824458] Lustre: lustre-OST0000: new disk, initializing [14959.827291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14959.857067] Lustre: 480799:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14959.946682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14967.012196] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14968.427218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14968.447261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14968.561586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14974.994055] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14984.172303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14984.173602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14984.184495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14984.200698] Lustre: Skipped 1 previous similar message [14984.225059] LustreError: Skipped 1 previous similar message [14987.835628] Lustre: server umount lustre-OST0000 complete [14994.401697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14994.404212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14994.414273] Lustre: Skipped 1 previous similar message [14994.424589] Lustre: Skipped 1 previous similar message [14999.520902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14999.529026] Lustre: Skipped 2 previous similar messages [15004.643104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15004.656042] Lustre: Skipped 1 previous similar message [15006.175370] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15006.315992] Lustre: server umount lustre-MDT0000 complete [15010.056276] LustreError: 480798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125238 with bad export cookie 10290294249618909177 [15010.059647] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15010.085735] LustreError: 480798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15010.505788] Lustre: server umount lustre-MDT0001 complete [15022.499751] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [15025.296134] Key type lgssc unregistered [15025.560337] LNet: 482573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15025.581403] LNetError: 482573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15025.600300] LNet: Removed LNI 192.168.204.144@tcp [15026.503641] Key type .llcrypt unregistered [15026.508668] Key type ._llcrypt unregistered [15049.665498] Key type ._llcrypt registered [15049.668485] Key type .llcrypt registered [15049.788935] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [15051.155355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15051.192075] alg: No test for adler32 (adler32-zlib) [15052.299706] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [15052.538142] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [15054.271138] Key type lgssc registered [15055.746818] Lustre: Echo OBD driver; http://www.lustre.org/ [15058.686855] alg: No test for zstdfast (zstdfast-lustre-generic) [15058.704726] alg: No test for zstdfast (zstdfast-lustre-scomp) [15064.828185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15064.846613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15066.495386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15071.598278] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15076.835140] LustreError: 483912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15076.866373] LustreError: 483912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15080.359657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15080.711419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15085.302637] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15092.106736] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15096.449207] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15103.635860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15104.048776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15110.895773] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15117.759439] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15128.878085] loop0: detected capacity change from 0 to 204800000 [15129.078654] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15129.098681] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15129.115441] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15135.100565] loop0: detected capacity change from 0 to 204800000 [15135.315252] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15135.335306] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15135.348933] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15140.647940] loop0: detected capacity change from 0 to 204800000 [15146.702468] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15146.848964] Lustre: fs15246-OST0000: new disk, initializing [15146.852820] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15146.863854] Lustre: 487169:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15146.926560] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15153.931411] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15165.491761] loop1: detected capacity change from 0 to 204800000 [15172.312534] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15172.482386] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15172.541353] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15172.684514] Lustre: fs15246-MDT0000: new disk, initializing [15172.900078] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15172.931346] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15173.091469] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15174.131778] Lustre: 488274:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000093b4b17b x1876033830698752/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789125418 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15177.864972] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15179.238577] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15179.248202] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15179.285170] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15198.182866] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15198.202187] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15203.309436] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15204.455948] Lustre: server umount fs15246-MDT0000 complete [15208.420026] LustreError: 483908:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15212.980812] Lustre: server umount fs15246-OST0000 complete [15223.785487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15223.795886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15223.799580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15223.813896] Lustre: Skipped 1 previous similar message [15228.833373] Lustre: server umount lustre-OST0000 complete [15234.016666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15234.040366] Lustre: Skipped 1 previous similar message [15234.049823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15239.137610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15239.149756] Lustre: Skipped 2 previous similar messages [15239.751820] Lustre: server umount lustre-MDT0000 complete [15243.232479] LustreError: 483912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15243.242063] LustreError: 483912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15244.300945] LustreError: 483937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125473 with bad export cookie 3140759023896807552 [15244.303785] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15244.307246] LustreError: 483937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15244.589948] Lustre: server umount lustre-MDT0001 complete [15254.513716] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [15256.957888] Key type lgssc unregistered [15257.301811] LNet: 491337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15257.307960] LNetError: 491337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15258.365691] LNet: Removed LNI 192.168.204.144@tcp [15259.239327] Key type .llcrypt unregistered [15259.241512] Key type ._llcrypt unregistered [15279.916203] Key type ._llcrypt registered [15279.919288] Key type .llcrypt registered [15280.017436] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [15280.928583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15280.956842] alg: No test for adler32 (adler32-zlib) [15282.061666] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [15282.318155] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [15284.023242] Key type lgssc registered [15285.261303] Lustre: Echo OBD driver; http://www.lustre.org/ [15288.316588] alg: No test for zstdfast (zstdfast-lustre-generic) [15288.373820] alg: No test for zstdfast (zstdfast-lustre-scomp) [15325.019285] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [15334.171737] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [15368.067751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15368.087407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15369.293942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15369.356277] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15369.415326] Lustre: lustre-MDT0000: new disk, initializing [15369.470949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15369.483917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15375.356374] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15387.213469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15387.368513] Lustre: 495848:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15387.432649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15387.435522] Lustre: Skipped 1 previous similar message [15387.549336] Lustre: lustre-MDT0001: new disk, initializing [15387.652565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15387.693776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15387.714637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15392.675797] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15401.080847] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15405.260599] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15412.393593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15412.620669] Lustre: lustre-OST0000: new disk, initializing [15412.624637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15412.630617] Lustre: 497591:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15412.748365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15418.342947] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15420.999336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15421.008312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15421.112203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15424.945883] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15428.904447] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15429.110196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15434.080643] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15434.271708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15436.772125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15436.772307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15436.775913] LustreError: Skipped 1 previous similar message [15436.777107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15436.809450] Lustre: Skipped 1 previous similar message [15441.892741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15441.902965] Lustre: Skipped 2 previous similar messages [15447.008722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15447.024019] Lustre: Skipped 1 previous similar message [15450.079605] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15450.323677] Lustre: server umount lustre-OST0000 complete [15457.249291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15457.258200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15457.281186] Lustre: Skipped 1 previous similar message [15460.632937] Lustre: server umount lustre-MDT0000 complete [15465.334698] LustreError: 495841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125694 with bad export cookie 349147766853194991 [15465.344811] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15465.356355] LustreError: 495841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15465.895913] Lustre: server umount lustre-MDT0001 complete [15497.398818] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [15507.600464] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [15541.056831] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15222 sec ========= 07:22:48 (1789125768) [15542.524734] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:22:49 (1789125769) === [15546.132564] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:22:53 (1789125773) === [15583.348193] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [15586.349877] Key type lgssc unregistered [15586.629562] LNet: 504051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15586.638866] LNetError: 504051:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15586.670240] LNet: Removed LNI 192.168.204.144@tcp [15587.605881] Key type .llcrypt unregistered [15587.608504] Key type ._llcrypt unregistered