[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 522524967 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003182] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008273] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011017] pid_max: default: 32768 minimum: 301 [ 0.013126] LSM: Security Framework initializing [ 0.014056] Yama: becoming mindful. [ 0.015062] SELinux: Initializing. [ 0.016077] *** VALIDATE selinux *** [ 0.025218] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030259] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033109] *** VALIDATE tmpfs *** [ 0.034512] *** VALIDATE proc *** [ 0.035275] *** VALIDATE cgroup *** [ 0.036014] *** VALIDATE cgroup2 *** [ 0.037283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041015] Speculative Store Bypass: Vulnerable [ 0.044359] debug: unmapping init [mem 0xffffffffb0e59000-0xffffffffb0e60fff] [ 0.046186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047662] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051018] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055272] rcu: Hierarchical SRCU implementation. [ 0.057394] smp: Bringing up secondary CPUs ... [ 0.058654] x86: Booting SMP configuration: [ 0.059038] .... node #0, CPUs: #1 #2 #3 [ 0.062546] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127024] node 0 deferred pages initialised in 58ms [ 0.131176] devtmpfs: initialized [ 0.132261] x86/mm: Memory block size: 128MB [ 0.134724] gcov: version magic: 0x41383552 [ 0.136160] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.137100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.138289] pinctrl core: initialized pinctrl subsystem [ 0.139204] [ 0.139937] ************************************************************* [ 0.140015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.141015] ** ** [ 0.142017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.143013] ** ** [ 0.144012] ** This means that this kernel is built to expose internal ** [ 0.145015] ** IOMMU data structures, which may compromise security on ** [ 0.146035] ** your system. ** [ 0.147018] ** ** [ 0.148014] ** If you see this message and you are not debugging the ** [ 0.149016] ** kernel, report this immediately to your vendor! ** [ 0.150014] ** ** [ 0.151018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152013] ************************************************************* [ 0.153694] NET: Registered protocol family 16 [ 0.154545] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.155086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.156067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.158011] cpuidle: using governor menu [ 0.160151] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.162633] PCI: Using configuration type 1 for base access [ 0.165123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.174126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.175037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176189] cryptd: max_cpu_qlen set to 1000 [ 0.178318] ACPI: Added _OSI(Module Device) [ 0.179014] ACPI: Added _OSI(Processor Device) [ 0.180016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.181016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.185613] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.189304] ACPI: Interpreter enabled [ 0.190059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.191016] ACPI: Using IOAPIC for interrupt routing [ 0.192126] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.193467] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.203185] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.204055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.205022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.206118] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.208467] acpiphp: Slot [2] registered [ 0.209112] acpiphp: Slot [5] registered [ 0.211242] acpiphp: Slot [6] registered [ 0.213250] acpiphp: Slot [7] registered [ 0.215260] acpiphp: Slot [8] registered [ 0.217173] acpiphp: Slot [9] registered [ 0.219167] acpiphp: Slot [10] registered [ 0.221149] acpiphp: Slot [3] registered [ 0.222188] acpiphp: Slot [4] registered [ 0.224131] acpiphp: Slot [11] registered [ 0.226341] acpiphp: Slot [12] registered [ 0.228253] acpiphp: Slot [13] registered [ 0.230245] acpiphp: Slot [14] registered [ 0.233159] acpiphp: Slot [15] registered [ 0.235220] acpiphp: Slot [16] registered [ 0.236194] acpiphp: Slot [17] registered [ 0.238204] acpiphp: Slot [18] registered [ 0.240108] acpiphp: Slot [19] registered [ 0.242128] acpiphp: Slot [20] registered [ 0.244106] acpiphp: Slot [21] registered [ 0.245136] acpiphp: Slot [22] registered [ 0.247206] acpiphp: Slot [23] registered [ 0.249194] acpiphp: Slot [24] registered [ 0.251159] acpiphp: Slot [25] registered [ 0.253183] acpiphp: Slot [26] registered [ 0.254110] acpiphp: Slot [27] registered [ 0.256110] acpiphp: Slot [28] registered [ 0.258185] acpiphp: Slot [29] registered [ 0.260147] acpiphp: Slot [30] registered [ 0.262110] acpiphp: Slot [31] registered [ 0.264100] PCI host bridge to bus 0000:00 [ 0.265021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.271033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.274057] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.278032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.281029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.284210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.288162] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.292378] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.303959] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.308022] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.311035] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.314021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.316027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.320197] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.322828] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.326078] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.329997] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.336017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.351018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.355758] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.361635] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.370018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.377017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.394031] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.404116] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.415016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.421026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.440019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.453596] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.460018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.467018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.485024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.495761] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.503027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.509030] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.525024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.535709] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.542018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.548015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.563018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.573481] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.578021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.585024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.599034] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.611607] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.614404] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.618433] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.621327] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.623232] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.629110] iommu: Default domain type: Passthrough [ 0.631713] SCSI subsystem initialized [ 0.633237] ACPI: bus type USB registered [ 0.635131] usbcore: registered new interface driver usbfs [ 0.637162] usbcore: registered new interface driver hub [ 0.639128] usbcore: registered new device driver usb [ 0.641203] pps_core: LinuxPPS API ver. 1 registered [ 0.643022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.646097] PTP clock support registered [ 0.650061] EDAC MC: Ver: 3.0.0 [ 0.652251] PCI: Using ACPI for IRQ routing [ 0.655220] NetLabel: Initializing [ 0.656019] NetLabel: domain hash size = 128 [ 0.658021] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.661138] NetLabel: unlabeled traffic allowed by default [ 0.664046] vgaarb: loaded [ 0.666087] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.667015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.671286] clocksource: Switched to clocksource kvm-clock [ 0.777549] VFS: Disk quotas dquot_6.6.0 [ 0.779171] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.782252] *** VALIDATE ramfs *** [ 0.783836] *** VALIDATE hugetlbfs *** [ 0.785702] pnp: PnP ACPI init [ 0.788310] pnp: PnP ACPI: found 6 devices [ 0.806140] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.810304] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.812506] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.814971] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.818030] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.820623] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.824025] NET: Registered protocol family 2 [ 0.826709] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.832264] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.836257] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.841897] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.845295] TCP: Hash tables configured (established 65536 bind 65536) [ 0.848641] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.852103] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854977] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.858136] NET: Registered protocol family 1 [ 0.861459] RPC: Registered named UNIX socket transport module. [ 0.863688] RPC: Registered udp transport module. [ 0.865891] RPC: Registered tcp transport module. [ 0.868705] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.871657] NET: Registered protocol family 44 [ 0.873883] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.876054] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.878144] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.880614] PCI: CLS 0 bytes, default 64 [ 0.882201] Unpacking initramfs... [ 2.394561] debug: unmapping init [mem 0xffff8e56bcc54000-0xffff8e56bffbffff] [ 2.398807] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.401334] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.404640] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.915906] Initialise system trusted keyrings [ 2.918144] Key type blacklist registered [ 2.921672] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.931881] zbud: loaded [ 2.935700] *** VALIDATE nfs *** [ 2.937191] *** VALIDATE nfs4 *** [ 2.939188] pstore: using deflate compression [ 2.946152] Platform Keyring initialized [ 3.052630] NET: Registered protocol family 38 [ 3.054827] Key type asymmetric registered [ 3.056689] Asymmetric key parser 'x509' registered [ 3.058954] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.062135] io scheduler mq-deadline registered [ 3.064087] io scheduler kyber registered [ 3.065908] io scheduler bfq registered [ 3.067899] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.071292] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.074051] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.077569] ACPI: Power Button [PWRF] [ 3.083871] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.090557] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.147439] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.156295] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.177502] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.205768] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.234274] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.239873] Non-volatile memory driver v1.3 [ 3.241668] Linux agpgart interface v0.103 [ 3.275735] virtio_blk virtio1: [vda] 142768 512-byte logical blocks (73.1 MB/69.7 MiB) [ 3.279366] vda: detected capacity change from 0 to 73097216 [ 3.295590] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.299203] vdb: detected capacity change from 0 to 1073741824 [ 3.314968] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.318341] vdc: detected capacity change from 0 to 2621440000 [ 3.336664] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.342316] vdd: detected capacity change from 0 to 2621440000 [ 3.358273] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.361941] vde: detected capacity change from 0 to 4294967296 [ 3.377144] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.380173] vdf: detected capacity change from 0 to 4294967296 [ 3.387297] libphy: Fixed MDIO Bus: probed [ 3.392191] usbcore: registered new interface driver usbserial_generic [ 3.394694] usbserial: USB Serial support registered for generic [ 3.396894] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.401221] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.403195] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.405868] mousedev: PS/2 mouse device common for all mice [ 3.408252] rtc_cmos 00:05: RTC can wake from S4 [ 3.410841] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.414146] rtc_cmos 00:05: registered as rtc0 [ 3.417342] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.417867] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.421060] intel_pstate: CPU model not supported [ 3.427394] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.430976] hid: raw HID events driver (C) Jiri Kosina [ 3.433569] usbcore: registered new interface driver usbhid [ 3.435971] usbhid: USB HID core driver [ 3.437866] drop_monitor: Initializing network drop monitor service [ 3.440972] Initializing XFRM netlink socket [ 3.443351] NET: Registered protocol family 10 [ 3.446394] Segment Routing with IPv6 [ 3.448085] NET: Registered protocol family 17 [ 3.450563] mpls_gso: MPLS GSO support [ 3.456583] RAS: Correctable Errors collector initialized. [ 3.458855] AVX version of gcm_enc/dec engaged. [ 3.460522] AES CTR mode by8 optimization enabled [ 3.539952] sched_clock: Marking stable (3539927821, 0)->(4614570883, -1074643062) [ 3.543764] registered taskstats version 1 [ 3.545613] Loading compiled-in X.509 certificates [ 3.548055] zswap: loaded using pool lzo/zbud [ 3.580430] Key type big_key registered [ 3.594192] Key type encrypted registered [ 3.596374] ima: No TPM chip found, activating TPM-bypass! [ 3.598883] ima: Allocated hash algorithm: sha1 [ 3.600880] ima: No architecture policies found [ 3.603017] evm: Initialising EVM extended attributes: [ 3.605219] evm: security.selinux [ 3.606563] evm: security.ima [ 3.607887] evm: security.capability [ 3.609366] evm: HMAC attrs: 0x1 [ 3.612193] rtc_cmos 00:05: setting system clock to 2026-09-11 07:03:37 UTC (1789110217) [ 3.619347] debug: unmapping init [mem 0xffffffffb1e03000-0xffffffffb1ffffff] [ 3.622981] debug: unmapping init [mem 0xffffffffb0b82000-0xffffffffb0e58fff] [ 3.632088] Write protecting the kernel read-only data: 28672k [ 3.636052] debug: unmapping init [mem 0xffffffffaf203000-0xffffffffaf3fffff] [ 3.639433] debug: unmapping init [mem 0xffffffffafb14000-0xffffffffafbfffff] [ 3.678266] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.686097] systemd[1]: Detected virtualization kvm. [ 3.688225] systemd[1]: Detected architecture x86-64. [ 3.690350] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.714992] systemd[1]: No hostname configured. [ 3.717218] systemd[1]: Set hostname to . [ 3.719749] random: systemd: uninitialized urandom read (16 bytes read) [ 3.722876] systemd[1]: Initializing machine ID from random generator. [ 3.765445] random: ln: uninitialized urandom read (6 bytes read) [ 3.868153] random: systemd: uninitialized urandom read (16 bytes read) [ 3.871815] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.882576] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 3.887188] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.533388] device-mapper: uevent: version 1.0.3 [ 4.535839] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.355235] random: fast init done [ 5.399809] virtio_net virtio0 ens2: renamed from eth0 [ 5.544845] scsi host0: ata_piix [ 5.558686] scsi host1: ata_piix [ 5.560478] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.563309] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.252583] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.115544] random: crng init done [ 10.117488] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.653980] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.839399] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.089437] SELinux: Disabled at runtime. [ 12.146669] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.155854] systemd[1]: Detected virtualization kvm. [ 12.157907] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.653378] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.657066] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.661948] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.668375] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.672366] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.679727] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.685433] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ 12.863792] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.190387] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.539862] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.549597] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.729629] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.740728] EDAC sbridge: Ver: 1.1.2 [ 15.533260] Key type dns_resolver registered [ 15.860660] NFS: Registering the id_resolver key type [ 15.862745] Key type id_resolver registered [ 15.864338] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg349-server login: [ 91.417196] hrtimer: interrupt took 7177343 ns [ 106.771572] libcfs: loading out-of-tree module taints kernel. [ 106.848300] Key type ._llcrypt registered [ 106.851633] Key type .llcrypt registered [ 106.978394] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 129.417209] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 131.296639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 131.309989] alg: No test for adler32 (adler32-zlib) [ 133.101915] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 134.232486] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 136.120063] Key type lgssc registered [ 138.533471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.075834] alg: No test for zstdfast (zstdfast-lustre-generic) [ 145.136850] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 157.245694] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 203.590911] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 221.679570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 221.723502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.033863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 223.153081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 223.340981] Lustre: lustre-MDT0000: new disk, initializing [ 223.688688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.761961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 230.403724] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.894735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 244.004504] Lustre: 6596:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 244.052923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 244.058077] Lustre: Skipped 1 previous similar message [ 244.148570] Lustre: lustre-MDT0001: new disk, initializing [ 244.255173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.296366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 244.314086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 249.015328] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.956866] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 267.941375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 268.481269] Lustre: lustre-OST0000: new disk, initializing [ 268.515799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 268.560923] Lustre: 8501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 268.741066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 274.498547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 274.516789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 274.650000] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 276.853858] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 294.164946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 294.276301] Lustre: lustre-OST0001: new disk, initializing [ 294.279822] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 294.285665] Lustre: 9558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 294.358927] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 301.887491] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 304.249284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 304.260722] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 304.323510] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 314.796749] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 323.845823] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 330.153363] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing check_logdir /tmp/testlogs/ [ 335.053971] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing yml_node [ 340.586494] Lustre: DEBUG MARKER: Client: 2.17.53.26 [ 343.601843] Lustre: DEBUG MARKER: MDS: 2.17.53.26 [ 346.499866] Lustre: DEBUG MARKER: OSS: 2.17.53.26 [ 348.047737] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 03:09:19 EDT 2026 [ 366.096386] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 367.537910] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 386.527844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.540901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 386.554668] Lustre: Skipped 1 previous similar message [ 389.592777] Lustre: server umount lustre-MDT0000 complete [ 394.075526] LustreError: 6588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789110607 with bad export cookie 9935514005291116767 [ 394.081244] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 394.095346] LustreError: 6588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 396.770399] LustreError: 6602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.772641] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.784742] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 396.784751] Lustre: Skipped 3 previous similar messages [ 396.814749] LustreError: 6602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 396.879903] Lustre: Skipped 3 previous similar messages [ 399.787098] LustreError: 6604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.787400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 399.815631] LustreError: 6604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 399.823363] Lustre: Skipped 2 previous similar messages [ 400.517136] Lustre: server umount lustre-MDT0001 complete [ 404.934175] Lustre: server umount lustre-OST0000 complete [ 409.507342] Lustre: server umount lustre-OST0001 complete [ 417.986301] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 430.043284] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 477.750131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.128247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 478.183979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 478.369831] Lustre: lustre-MDT0000: new disk, initializing [ 478.508247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.536467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 482.581522] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 492.717309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.860381] Lustre: 15091:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 492.943422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 492.952281] Lustre: Skipped 1 previous similar message [ 493.052512] Lustre: lustre-MDT0001: new disk, initializing [ 493.174815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 493.230117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 493.245799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 497.843413] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 505.888739] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 510.142918] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 517.481362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 518.066528] Lustre: lustre-OST0000: new disk, initializing [ 518.073931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 518.087092] Lustre: 16830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 518.248821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.026632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 520.044361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 520.335726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 526.664770] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 535.530266] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 540.671231] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 540.987055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 545.973589] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 546.299108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 550.882049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 550.890118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.899058] LustreError: Skipped 1 previous similar message [ 550.901064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 550.936136] Lustre: Skipped 1 previous similar message [ 554.589228] Lustre: server umount lustre-OST0000 complete [ 559.589032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 559.604179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.628348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 559.632487] Lustre: Skipped 1 previous similar message [ 565.265357] Lustre: server umount lustre-MDT0000 complete [ 566.243779] LustreError: 15096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.261262] LustreError: 15096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 569.738770] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789110783 with bad export cookie 9935514005291118300 [ 569.758195] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.766271] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 570.107699] Lustre: server umount lustre-MDT0001 complete [ 578.474590] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 03:13:09 (1789110789) [ 600.250136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.033281] LustreError: 19562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.102639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.765809] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 606.177472] LustreError: 19563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.394862] LustreError: 19562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.428961] LustreError: 19562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 615.681872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.328726] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 627.737899] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 632.857163] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 635.392737] Lustre: Failing over lustre-MDT0000 [ 635.790815] Lustre: server umount lustre-MDT0000 complete [ 636.896612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.900889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 636.921829] Lustre: Skipped 1 previous similar message [ 636.936482] LustreError: 20273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.257230] Lustre: 3679:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789110850/real 1789110850] req@ffff8e572fbe1500 x1876018172425728/t0(0) o400->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789110866 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 652.293221] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.870154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.497556] LustreError: 3678:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e56074fc380 x1876018172428672/t0(0) o250->MGC192.168.203.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 662.983982] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 668.130575] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 668.149231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 668.169719] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 668.947580] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 679.131934] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 687.589864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.596213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 687.611573] Lustre: Skipped 1 previous similar message [ 687.621057] Lustre: Skipped 2 previous similar messages [ 698.335126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 698.526715] Lustre: server umount lustre-MDT0000 complete [ 698.857054] LustreError: 20273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 698.884329] LustreError: 20273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 703.647748] LustreError: 19544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789110917 with bad export cookie 9935514005291119714 [ 703.667314] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.680903] LustreError: 19544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 704.095732] Lustre: server umount lustre-MDT0001 complete [ 714.459276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.996385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.004658] Lustre: Skipped 2 previous similar messages [ 720.024433] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 730.119030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.752598] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 742.332304] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 747.276365] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 749.718519] Lustre: Failing over lustre-MDT0000 [ 750.205780] Lustre: server umount lustre-MDT0000 complete [ 751.072981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.087282] Lustre: Skipped 1 previous similar message [ 765.415303] LustreError: 22559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 765.448391] LustreError: 22559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 766.431128] Lustre: 3680:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789110964/real 1789110964] req@ffff8e572e427480 x1876018172456320/t0(0) o400->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789110980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 766.501876] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 772.470620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.678887] LustreError: 3678:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e5737dd8700 x1876018172459392/t0(0) o250->MGC192.168.203.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 777.137345] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 782.304797] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 782.335838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 782.344968] Lustre: Skipped 1 previous similar message [ 782.385843] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 783.591276] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 792.816692] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 797.664569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 797.691593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 797.730296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 797.733394] Lustre: Skipped 5 previous similar messages [ 803.715277] Lustre: server umount lustre-MDT0000 complete [ 807.926409] LustreError: 22542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789111021 with bad export cookie 9935514005291120652 [ 807.930518] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 807.942640] LustreError: 22542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 808.241471] Lustre: server umount lustre-MDT0001 complete [ 818.604883] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 03:17:09 (1789111029) [ 840.931905] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 852.621459] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 899.761923] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 912.032415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.387223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 912.451652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 912.563677] Lustre: lustre-MDT0000: new disk, initializing [ 912.697667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.709782] Lustre: Skipped 2 previous similar messages [ 912.733522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 917.685945] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 930.727434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.980755] Lustre: 28598:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 931.344620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 935.744653] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 943.501486] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 948.480499] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 956.181737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 956.589674] Lustre: 30342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 958.287623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 958.357619] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 958.451515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 958.473893] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 963.295926] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 971.320945] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 976.657251] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 999.393554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.399173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 999.405398] Lustre: Skipped 1 previous similar message [ 999.410229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.422282] Lustre: Skipped 3 previous similar messages [ 1010.143695] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1010.394442] Lustre: server umount lustre-OST0000 complete [ 1020.619546] Lustre: server umount lustre-MDT0000 complete [ 1024.739731] LustreError: 28590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789111238 with bad export cookie 9935514005291121058 [ 1024.740874] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1024.751470] LustreError: 28590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1025.164929] Lustre: server umount lustre-MDT0001 complete [ 1037.366565] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1040.647599] Key type lgssc unregistered [ 1041.003688] LNet: 32193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.019285] LNetError: 32193:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.041942] LNet: Removed LNI 192.168.203.149@tcp [ 1042.034190] Key type .llcrypt unregistered [ 1042.036361] Key type ._llcrypt unregistered [ 1070.838392] Key type ._llcrypt registered [ 1070.840669] Key type .llcrypt registered [ 1070.983824] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1072.705849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1072.879366] alg: No test for adler32 (adler32-zlib) [ 1074.197807] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 1074.509549] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1076.231969] Key type lgssc registered [ 1078.026597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1082.129978] alg: No test for zstdfast (zstdfast-lustre-generic) [ 1082.168260] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 1087.646665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1087.688515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.324928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.257422] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1099.757861] LustreError: 33675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.790582] LustreError: 33675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1103.209429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.611507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1108.217643] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1113.797550] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1117.992874] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1125.968281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1126.505618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.576970] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1134.486772] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1135.755752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1139.928473] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1155.727351] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 03:22:46 (1789111366) [ 1158.082426] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1160.099374] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1167.563872] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 03:22:58 (1789111378) [ 1177.113887] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 03:23:08 (1789111388) [ 1178.078957] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1186.609979] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1186.615455] Lustre: Skipped 2 previous similar messages [ 1188.379938] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 03:23:19 (1789111399) [ 1190.873956] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1190.880357] Lustre: Skipped 2 previous similar messages [ 1200.040652] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1200.055825] Lustre: Skipped 4 previous similar messages [ 1216.593112] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 03:23:48 (1789111428) [ 1219.853446] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1223.784648] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1233.051817] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 03:24:04 (1789111444) [ 1234.980777] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1234.995151] Lustre: Skipped 6 previous similar messages [ 1249.088420] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 03:24:19 (1789111459) [ 1267.170066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1267.182659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.207110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1269.216285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1269.222817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1269.233757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.256811] Lustre: Skipped 1 previous similar message [ 1271.001283] Lustre: server umount lustre-OST0000 complete [ 1277.924143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1277.943919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.981550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1281.495435] Lustre: server umount lustre-MDT0000 complete [ 1284.576492] LustreError: 34387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.601245] LustreError: 34387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1285.074336] LustreError: 33657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789111498 with bad export cookie 18035686955189605069 [ 1285.083142] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1285.091511] LustreError: 33657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1285.642180] Lustre: server umount lustre-MDT0001 complete [ 1297.781655] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1300.949907] Key type lgssc unregistered [ 1301.253445] LNet: 41176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.273644] LNetError: 41176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.305801] LNet: Removed LNI 192.168.203.149@tcp [ 1302.265115] Key type .llcrypt unregistered [ 1302.270037] Key type ._llcrypt unregistered [ 1322.127178] Key type ._llcrypt registered [ 1322.130123] Key type .llcrypt registered [ 1322.273391] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 1339.865194] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1340.765978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1340.890981] alg: No test for adler32 (adler32-zlib) [ 1342.134050] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 1342.430451] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1344.111155] Key type lgssc registered [ 1345.187330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1348.093936] alg: No test for zstdfast (zstdfast-lustre-generic) [ 1348.121852] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 1380.403626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1380.430883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.701585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1381.732119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1381.841506] Lustre: lustre-MDT0000: new disk, initializing [ 1381.966825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.999038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1386.351546] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1397.229026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.343756] Lustre: 44379:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1397.392245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1397.396625] Lustre: Skipped 1 previous similar message [ 1397.467336] Lustre: lustre-MDT0001: new disk, initializing [ 1397.574658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1397.607418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1397.626627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1402.139012] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1409.248481] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1412.657987] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1420.654802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1420.988090] Lustre: lustre-OST0000: new disk, initializing [ 1420.998253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1421.003217] Lustre: 46121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1421.120141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.973725] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1430.626467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1430.638565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1430.734040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1434.362807] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1439.060345] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1439.245223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1443.537384] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1443.825895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1446.379097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1446.386962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.390631] LustreError: Skipped 1 previous similar message [ 1446.392678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1446.422784] Lustre: Skipped 1 previous similar message [ 1451.501544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1451.515115] Lustre: Skipped 1 previous similar message [ 1452.620438] Lustre: server umount lustre-OST0000 complete [ 1456.608238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.612520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.639739] Lustre: Skipped 1 previous similar message [ 1456.648867] Lustre: Skipped 1 previous similar message [ 1461.730364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.738289] Lustre: Skipped 2 previous similar messages [ 1462.167250] Lustre: server umount lustre-MDT0000 complete [ 1465.692141] LustreError: 44370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789111679 with bad export cookie 3166449609826001967 [ 1465.692974] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1465.711233] LustreError: 44370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1466.092421] Lustre: server umount lustre-MDT0001 complete [ 1475.417626] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 03:28:07 (1789111687) [ 1482.389384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.056752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.209392] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1488.357275] LustreError: 48430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.367772] LustreError: 48430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1492.449895] LustreError: 48431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1495.044930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1495.356745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1499.055471] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1504.044965] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1507.130986] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1513.476839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1513.874458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1519.413421] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1524.660990] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1533.302946] LustreError: 50618:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1534.157987] Lustre: Modifying parameter general.timeout=1 in log params [ 1535.743645] Lustre: Modifying parameter general.timeout=3 in log params [ 1535.748445] Lustre: Skipped 1 previous similar message [ 1538.465370] Lustre: Modifying parameter general.timeout=6 in log params [ 1538.473043] Lustre: Skipped 2 previous similar messages [ 1542.789501] Lustre: Modifying parameter general.timeout=11 in log params [ 1542.797175] Lustre: Skipped 4 previous similar messages [ 1551.523428] Lustre: Modifying parameter general.timeout=21 in log params [ 1551.527809] Lustre: Skipped 9 previous similar messages [ 1568.274135] Lustre: Modifying parameter general.timeout=39 in log params [ 1568.278210] Lustre: Skipped 17 previous similar messages [ 1600.283680] Lustre: Modifying parameter general.timeout=77 in log params [ 1600.290535] Lustre: Skipped 37 previous similar messages [ 1655.699996] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 03:31:06 (1789111866) [ 1658.780853] Lustre: 57556:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1660.338333] Lustre: 57654:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1664.048316] Lustre: 57854:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1664.073563] Lustre: 57854:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1664.697205] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1664.703078] Lustre: Skipped 58 previous similar messages [ 1693.586210] Lustre: 59432:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1693.593792] Lustre: 59432:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1693.614971] LustreError: 59432:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1697.452761] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 03:31:49 (1789111909) [ 1710.563959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1710.581543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.607269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1713.128534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1713.150902] Lustre: Skipped 1 previous similar message [ 1715.618568] Lustre: server umount lustre-MDT0000 complete [ 1718.248246] LustreError: 58331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1718.271500] LustreError: 58331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1719.002556] LustreError: 48410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789111932 with bad export cookie 3166449609826002849 [ 1719.002912] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1719.012115] LustreError: 48410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1719.232917] Lustre: server umount lustre-MDT0001 complete [ 1733.040850] Lustre: server umount lustre-OST0000 complete [ 1757.685290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.921654] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1758.236513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.090891] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1769.627350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1769.720294] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1769.787592] Lustre: 61962:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1773.783142] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1778.596253] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1781.456877] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1786.562473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1786.739078] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1792.394379] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1797.428833] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1798.910898] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1798.919191] Lustre: Skipped 17 previous similar messages [ 1798.957137] Lustre: 64189:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1804.256985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1804.263203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.272720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1804.275355] Lustre: Skipped 3 previous similar messages [ 1810.417285] Lustre: server umount lustre-OST0000 complete [ 1815.015938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.030455] Lustre: Skipped 1 previous similar message [ 1825.248369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.261606] Lustre: Skipped 8 previous similar messages [ 1828.319165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1828.519275] Lustre: server umount lustre-MDT0000 complete [ 1830.368722] LustreError: 62848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.381169] LustreError: 62848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1831.601838] LustreError: 61954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112045 with bad export cookie 3166449609826005845 [ 1831.606648] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1831.613788] LustreError: 61954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1831.864536] Lustre: server umount lustre-MDT0001 complete [ 1842.616894] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1845.669632] Key type lgssc unregistered [ 1845.968540] LNet: 65383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.974476] LNetError: 65383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.983552] LNet: Removed LNI 192.168.203.149@tcp [ 1846.780410] Key type .llcrypt unregistered [ 1846.783051] Key type ._llcrypt unregistered [ 1870.039881] Key type ._llcrypt registered [ 1870.042231] Key type .llcrypt registered [ 1870.146206] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1871.033365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1871.096013] alg: No test for adler32 (adler32-zlib) [ 1872.195777] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 1872.416379] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1874.119394] Key type lgssc registered [ 1875.129453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1877.649590] alg: No test for zstdfast (zstdfast-lustre-generic) [ 1877.659727] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 1883.043160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1883.065519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.649149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.247899] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1894.884064] LustreError: 66864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.910905] LustreError: 66864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1897.512577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.945366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1901.923919] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1906.633478] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1909.885079] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1915.436150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1915.925288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.695262] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1926.826300] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1939.395561] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1949.153942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1949.160080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.176704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.664911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.688080] Lustre: Skipped 1 previous similar message [ 1949.695916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.702864] Lustre: Skipped 1 previous similar message [ 1953.276992] Lustre: server umount lustre-MDT0000 complete [ 1954.784044] LustreError: 66863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1954.819819] LustreError: 66863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1957.415401] LustreError: 66843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112171 with bad export cookie 11087446932548646332 [ 1957.418104] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1957.428553] LustreError: 66843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1957.785312] Lustre: server umount lustre-MDT0001 complete [ 1971.912403] Lustre: server umount lustre-OST0000 complete [ 1996.115310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.420871] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1996.471538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1996.784948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.064826] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2008.883967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.949495] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2008.967891] Lustre: 71395:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2009.343644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2012.728160] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2017.845713] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2021.275393] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2027.048410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2027.228881] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2027.557924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2033.329833] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2038.541985] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2040.019331] Lustre: 73627:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2041.573235] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2047.468082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2047.475910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.490698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2052.500670] Lustre: server umount lustre-OST0000 complete [ 2058.210209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.223273] Lustre: Skipped 1 previous similar message [ 2058.226750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.229775] Lustre: Skipped 1 previous similar message [ 2062.403091] Lustre: server umount lustre-MDT0000 complete [ 2063.330073] LustreError: 71405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.361067] LustreError: 71405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2066.486923] LustreError: 72134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112280 with bad export cookie 11087446932548647158 [ 2066.487538] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.503099] LustreError: 72134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2066.868429] Lustre: server umount lustre-MDT0001 complete [ 2077.469447] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2080.469639] Key type lgssc unregistered [ 2080.800906] LNet: 74820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.811838] LNetError: 74820:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.824626] LNet: Removed LNI 192.168.203.149@tcp [ 2081.656422] Key type .llcrypt unregistered [ 2081.661678] Key type ._llcrypt unregistered [ 2104.214702] Key type ._llcrypt registered [ 2104.218405] Key type .llcrypt registered [ 2104.302553] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2105.113804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2105.245868] alg: No test for adler32 (adler32-zlib) [ 2106.346961] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 2106.579561] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2108.240280] Key type lgssc registered [ 2109.307900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2111.672944] alg: No test for zstdfast (zstdfast-lustre-generic) [ 2111.683813] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 2116.964149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2116.981113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.478854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.543644] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2128.865844] LustreError: 76299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.875050] LustreError: 76299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2130.206072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.710808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2134.532768] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2139.812651] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2143.564854] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2149.760725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2150.478920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.267824] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2162.021376] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2173.231593] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2174.338526] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2175.433222] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2177.800630] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2177.808473] Lustre: Skipped 1 previous similar message [ 2181.928623] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2181.940707] Lustre: Skipped 3 previous similar messages [ 2190.157517] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2190.164096] Lustre: Skipped 8 previous similar messages [ 2206.715722] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2206.719610] Lustre: Skipped 19 previous similar messages [ 2238.869856] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2238.881321] Lustre: Skipped 35 previous similar messages [ 2267.918516] Lustre: 84054:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2270.443627] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 03:41:22 (1789112482) [ 2271.708188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2273.217713] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 03:41:25 (1789112485) [ 2276.836351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2276.839316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2276.846092] Lustre: Skipped 1 previous similar message [ 2276.855446] Lustre: Skipped 1 previous similar message [ 2281.952975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2281.973828] Lustre: Skipped 1 previous similar message [ 2282.884506] Lustre: server umount lustre-OST0000 complete [ 2287.074822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.088846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.099784] Lustre: Skipped 1 previous similar message [ 2287.125061] Lustre: Skipped 1 previous similar message [ 2292.197188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.206080] Lustre: Skipped 1 previous similar message [ 2293.396958] Lustre: server umount lustre-MDT0000 complete [ 2297.318408] LustreError: 76298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.340645] LustreError: 76298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2297.365091] LustreError: 84056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112511 with bad export cookie 11040921142851136177 [ 2297.370928] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.374660] LustreError: 84056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2297.909586] Lustre: server umount lustre-MDT0001 complete [ 2310.661666] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2314.737925] Key type lgssc unregistered [ 2315.193682] LNet: 85288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.203602] LNetError: 85288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.262431] LNet: Removed LNI 192.168.203.149@tcp [ 2317.458103] Key type .llcrypt unregistered [ 2317.465102] Key type ._llcrypt unregistered [ 2324.241363] Key type ._llcrypt registered [ 2324.243574] Key type .llcrypt registered [ 2324.413429] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2331.269606] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules [ 2332.547905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2332.659787] alg: No test for adler32 (adler32-zlib) [ 2333.719470] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 2334.025781] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2335.735637] Key type lgssc registered [ 2336.678177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2338.197128] LustreError: 86537:0:(lu_object.c:1479:lu_context_key_register()) cfs_race id 60d sleeping [ 2356.361519] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2360.369778] LustreError: 86537:0:(lu_object.c:1479:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2361.100221] alg: No test for zstdfast (zstdfast-lustre-generic) [ 2361.187871] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 2366.389696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2366.407716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.881554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.883519] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2377.187776] LustreError: 87610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.201886] LustreError: 87610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2381.729116] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 03:43:13 (1789112593) [ 2382.325938] LustreError: 87628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.429641] LustreError: 87642:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2386.442263] LustreError: 87642:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2386.458629] LustreError: 87642:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2386.800187] Lustre: server umount lustre-MDT0000 complete [ 2400.795038] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2403.386037] Key type lgssc unregistered [ 2403.691745] LNet: 89104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.701683] LNetError: 89104:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2403.722552] LNet: Removed LNI 192.168.203.149@tcp [ 2404.597780] Key type .llcrypt unregistered [ 2404.600529] Key type ._llcrypt unregistered [ 2420.987195] Key type ._llcrypt registered [ 2420.989508] Key type .llcrypt registered [ 2421.097860] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2422.290135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2422.355058] alg: No test for adler32 (adler32-zlib) [ 2423.375851] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 2423.586821] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2425.271163] Key type lgssc registered [ 2426.277300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2428.897656] alg: No test for zstdfast (zstdfast-lustre-generic) [ 2428.918464] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 2433.397459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2433.413696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.898360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.369629] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2444.261458] LustreError: 90397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.282706] LustreError: 90397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2446.086207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.688000] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2450.547684] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2455.873521] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2459.503629] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2465.516976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2465.975794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2472.000556] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2477.315203] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2530.504361] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 03:45:41 (1789112741) [ 2532.855953] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2534.893831] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 03:45:46 (1789112746) [ 2542.048330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.058246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2542.068497] Lustre: Skipped 2 previous similar messages [ 2542.088276] Lustre: Skipped 2 previous similar messages [ 2544.930450] Lustre: server umount lustre-MDT0000 complete [ 2547.169191] LustreError: 92767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.187564] LustreError: 92767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2547.736300] LustreError: 90377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112761 with bad export cookie 13053909250402888236 [ 2547.744982] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2547.750401] LustreError: 90377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2548.007736] Lustre: server umount lustre-MDT0001 complete [ 2560.547936] Lustre: server umount lustre-OST0000 complete [ 2572.991862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.644944] LustreError: 94333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.772464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.154155] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2578.923193] LustreError: 94334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.035201] LustreError: 94333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.130859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.083036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2592.962275] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2598.191105] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2601.543751] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2614.703830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2614.975213] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2614.983429] LustreError: 94323:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2615.010191] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2615.025511] LustreError: 96192:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2615.035355] LustreError: 96192:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2615.044130] LustreError: 96192:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2615.152752] Lustre: server umount lustre-OST0000 complete [ 2615.155688] LustreError: 96192:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2625.784455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2626.044222] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2626.058486] LustreError: 94324:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2626.096218] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2626.109841] LustreError: 96660:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2626.122966] LustreError: 96660:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2626.126269] LustreError: 96660:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2626.208513] Lustre: server umount lustre-OST0000 complete [ 2626.212067] LustreError: 96660:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2636.748612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.956390] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2636.964522] Lustre: Found index 0 for lustre-OST0000, updating log [ 2636.984864] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2637.151292] Lustre: lustre-OST0000: new disk, initializing [ 2637.158122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2637.171154] Lustre: 97193:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2637.467581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.510498] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2648.369573] Lustre: Failing over lustre-OST0000 [ 2648.627392] Lustre: server umount lustre-OST0000 complete [ 2654.178250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2654.180395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.184150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2654.206741] Lustre: Skipped 1 previous similar message [ 2658.339207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2658.353483] Lustre: Skipped 2 previous similar messages [ 2662.574248] Lustre: server umount lustre-MDT0000 complete [ 2663.401707] LustreError: 94328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.420456] LustreError: 94328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2666.422928] LustreError: 94314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112880 with bad export cookie 13053909250402890987 [ 2666.422928] LustreError: 96212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789112880 with bad export cookie 13053909250402890987 [ 2666.430484] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.432548] LustreError: 96212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2666.749376] Lustre: server umount lustre-MDT0001 complete [ 2676.165592] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 03:48:07 (1789112887) [ 2693.398581] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2704.669591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.050958] LustreError: 99518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.246593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.169857] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2719.358732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2725.850657] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2729.367908] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2737.596821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2738.656805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.670592] Lustre: Skipped 1 previous similar message [ 2738.690716] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2744.832091] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2744.839759] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2744.858742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2746.363900] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2749.924376] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2749.938400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2750.949548] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2750.962525] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2750.966423] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2750.974364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2754.954227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2755.077484] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2755.190718] Lustre: lustre-OST0001: new disk, initializing [ 2755.196899] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2755.207739] Lustre: 101788:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2761.561696] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2761.822575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2761.837494] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2761.926968] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2768.356418] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2774.886647] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2779.165214] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.49@tcp (stopping) [ 2781.676696] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2781.687539] LustreError: Skipped 1 previous similar message [ 2781.690880] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.276201] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.49@tcp (stopping) [ 2784.289921] Lustre: Skipped 4 previous similar messages [ 2785.541555] Lustre: server umount lustre-MDT0001 complete [ 2787.812665] LustreError: 99515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.841191] LustreError: 99515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2795.610994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.753859] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2795.765914] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2795.820829] Lustre: 99507:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2797.749581] LustreError: 103351:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2797.756449] LustreError: 103351:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2797.768387] LustreError: 99510:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2797.929257] LustreError: 103358:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -17 [ 2797.948649] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2797.948649] [ 2801.508708] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2801.642142] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2812.330410] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 03:50:23 (1789113023) [ 2833.637158] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2852.875560] Lustre: 105363:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2873.232789] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2876.572768] Lustre: 106498:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2959.840924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2959.852064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.859221] Lustre: Skipped 2 previous similar messages [ 2959.878455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2963.157717] Lustre: server umount lustre-MDT0000 complete [ 2965.473584] LustreError: 99513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2965.493660] LustreError: 99513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2966.802451] LustreError: 99498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789113180 with bad export cookie 13053909250402891988 [ 2966.802777] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2966.820421] LustreError: 99498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2967.132614] Lustre: server umount lustre-MDT0001 complete [ 2981.125714] Lustre: server umount lustre-OST0000 complete [ 2994.702299] Lustre: server umount lustre-OST0001 complete [ 2999.862358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3008.533744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3016.325044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3025.900048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3044.453032] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3056.132469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.168543] Lustre: lustre-MDT0000: reset Object Index mappings [ 3056.866327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.869623] Lustre: Skipped 2 previous similar messages [ 3061.567764] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3073.155212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.177187] Lustre: lustre-MDT0001: reset Object Index mappings [ 3079.113399] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3082.134918] Lustre: 111556:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3088.993753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.316751] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3098.279706] LustreError: 111910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.293770] LustreError: 111910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3098.299241] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3101.379910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 3106.566389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3106.799684] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 3106.804197] Lustre: Skipped 5 previous similar messages [ 3112.437866] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3112.466443] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3114.897546] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3122.932670] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3126.970512] Lustre: 113402:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3175.844435] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 03:56:27 (1789113387) [ 3204.585884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.593077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.628334] Lustre: Skipped 6 previous similar messages [ 3204.654632] Lustre: Skipped 2 previous similar messages [ 3208.587156] Lustre: server umount lustre-MDT0000 complete [ 3212.714446] LustreError: 110425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789113426 with bad export cookie 13053909250402939105 [ 3212.726415] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3212.741971] LustreError: 110425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3214.817772] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.783663] Lustre: server umount lustre-MDT0001 complete [ 3231.028070] Lustre: server umount lustre-OST0000 complete [ 3249.119265] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3249.685863] Lustre: server umount lustre-OST0001 complete [ 3258.207396] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 3269.762624] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3304.793237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.068573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3305.095892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3305.187761] Lustre: lustre-MDT0000: new disk, initializing [ 3305.280976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3305.285910] Lustre: Skipped 3 previous similar messages [ 3305.308522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3310.114467] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3316.588981] Lustre: server umount lustre-MDT0000 complete [ 3325.967456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.371849] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3326.399761] Lustre: 117631:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3326.419345] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3331.305477] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3336.213095] Lustre: server umount lustre-MDT0000 complete [ 3343.977462] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 03:59:15 (1789113555) [ 3345.502573] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3347.383306] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 03:59:18 (1789113558) [ 3363.690791] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3372.614577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.022396] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3386.027764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.116872] Lustre: 119659:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3386.214661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3386.221547] Lustre: Skipped 1 previous similar message [ 3386.316576] Lustre: lustre-MDT0001: new disk, initializing [ 3386.428614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3386.444312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3390.382576] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3394.823546] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3400.809126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.218791] Lustre: lustre-OST0000: new disk, initializing [ 3401.226131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3401.238590] Lustre: 121209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3403.177380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3403.192167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3403.327222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3408.052415] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3417.982033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3418.146387] Lustre: lustre-OST0001: new disk, initializing [ 3418.149409] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3418.153968] Lustre: 122062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3419.546698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3419.559068] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3419.670157] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3424.481507] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3433.880402] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3438.528535] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3837.121972] Lustre: 128473:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3837.129214] Lustre: 128473:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3837.136370] Lustre: 128473:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3837.142111] Lustre: 128473:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3837.147604] Lustre: 128473:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3837.154407] Lustre: 128473:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4103.292200] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:11:54 (1789114314) [ 4121.059406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4121.069257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4121.083117] Lustre: Skipped 4 previous similar messages [ 4121.094202] Lustre: Skipped 8 previous similar messages [ 4125.320588] Lustre: server umount lustre-MDT0000 complete [ 4126.181198] LustreError: 119668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4126.210654] LustreError: 119668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 4129.058060] LustreError: 120397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789114342 with bad export cookie 13053909250403006046 [ 4129.058902] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4129.086190] LustreError: 120397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4131.299192] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4131.313259] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4131.322688] Lustre: Skipped 1 previous similar message [ 4131.344970] Lustre: Skipped 1 previous similar message [ 4135.560740] Lustre: server umount lustre-MDT0001 complete [ 4145.361045] Lustre: server umount lustre-OST0000 complete [ 4163.040244] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4163.278609] Lustre: server umount lustre-OST0001 complete [ 4169.621991] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 4179.118263] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4211.659051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4211.890479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4211.915463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4211.981991] Lustre: lustre-MDT0000: new disk, initializing [ 4212.050434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4212.054866] Lustre: Skipped 5 previous similar messages [ 4212.066160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4215.856420] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4225.510192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.597580] Lustre: 131505:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4225.646964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4225.653196] Lustre: Skipped 1 previous similar message [ 4225.747692] Lustre: lustre-MDT0001: new disk, initializing [ 4225.864987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4225.874810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4230.615472] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4237.798912] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4241.599516] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4248.172352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4248.428855] Lustre: lustre-OST0000: new disk, initializing [ 4248.432356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4248.444830] Lustre: 133248:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4248.537402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4248.542217] Lustre: Skipped 1 previous similar message [ 4249.899682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4249.913496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4250.017897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4254.566888] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4261.332803] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4270.151549] Lustre: lustre-MDD0000: changelog on [ 4272.960897] Lustre: lustre-MDD0001: changelog on [ 6017.568821] Lustre: lustre-MDD0001: changelog off [ 6019.002397] Lustre: 133997:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6019.709174] Lustre: lustre-MDD0000: changelog off [ 6024.674370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6024.686309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6024.692359] Lustre: Skipped 3 previous similar messages [ 6029.796918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6029.813772] Lustre: Skipped 1 previous similar message [ 6030.575402] Lustre: server umount lustre-OST0000 complete [ 6034.915576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6034.921495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6034.927181] Lustre: Skipped 2 previous similar messages [ 6040.032801] LustreError: 133997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6040.047528] LustreError: 133997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6040.120926] Lustre: server umount lustre-MDT0000 complete [ 6043.903875] LustreError: 131498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116257 with bad export cookie 13053909250403785664 [ 6043.908549] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6043.915303] LustreError: 131498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6044.257496] Lustre: server umount lustre-MDT0001 complete [ 6056.354547] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6059.101113] Key type lgssc unregistered [ 6059.325743] LNet: 142227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6059.335467] LNetError: 142227:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6059.355323] LNet: Removed LNI 192.168.203.149@tcp [ 6060.187245] Key type .llcrypt unregistered [ 6060.189535] Key type ._llcrypt unregistered [ 6079.284588] Key type ._llcrypt registered [ 6079.287459] Key type .llcrypt registered [ 6079.426841] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 6095.026694] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6095.811647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6095.963931] alg: No test for adler32 (adler32-zlib) [ 6097.084654] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 6097.240698] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6098.871204] Key type lgssc registered [ 6100.315164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6103.589592] alg: No test for zstdfast (zstdfast-lustre-generic) [ 6103.604218] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 6139.481053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6139.526506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6140.763484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6140.795777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6140.858763] Lustre: lustre-MDT0000: new disk, initializing [ 6141.000837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6141.023583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6145.510269] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6156.029489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6156.115599] Lustre: 145478:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6156.159141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6156.162713] Lustre: Skipped 1 previous similar message [ 6156.258841] Lustre: lustre-MDT0001: new disk, initializing [ 6156.360574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6156.384846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6156.393970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6160.909723] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6168.804243] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6172.309306] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6178.606758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6178.849224] Lustre: lustre-OST0000: new disk, initializing [ 6178.851913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6178.856914] Lustre: 147220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6178.920784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6185.079762] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6187.121870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6187.137261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6187.253063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6192.885873] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6198.264519] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_module obdecho/obdecho [ 6200.152426] LustreError: 148093:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6200.168686] LustreError: 148093:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6202.062481] LustreError: 148189:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6217.697953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6217.717392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.739924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6221.119719] Lustre: server umount lustre-MDT0000 complete [ 6224.612541] LustreError: 145470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116438 with bad export cookie 6739527268044906279 [ 6224.617404] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6224.643032] LustreError: 145470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6224.989321] Lustre: server umount lustre-MDT0001 complete [ 6238.391727] Lustre: server umount lustre-OST0000 complete [ 6258.834667] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 6269.185107] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6307.224111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.441143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6307.495381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6307.580315] Lustre: lustre-MDT0000: new disk, initializing [ 6307.677759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6307.712321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6312.407776] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6322.669511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.775548] Lustre: 151891:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6322.816566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6322.819675] Lustre: Skipped 1 previous similar message [ 6322.913212] Lustre: lustre-MDT0001: new disk, initializing [ 6323.000834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6323.032074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6323.048994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6327.245477] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6334.617554] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6337.632702] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6343.981515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6344.280148] Lustre: lustre-OST0000: new disk, initializing [ 6344.285401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6344.292276] Lustre: 153633:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6344.401105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6350.038172] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6353.984351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6353.990403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6354.128542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6356.634446] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6360.657159] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6360.822959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6364.861292] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6365.055839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6369.760221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6369.761637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.766960] LustreError: Skipped 1 previous similar message [ 6369.768321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6369.768326] Lustre: Skipped 3 previous similar messages [ 6369.800473] Lustre: Skipped 3 previous similar messages [ 6373.102330] Lustre: server umount lustre-OST0000 complete [ 6379.491118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6379.501253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6379.518158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6379.526073] Lustre: Skipped 1 previous similar message [ 6382.720239] Lustre: server umount lustre-MDT0000 complete [ 6385.133499] LustreError: 151898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6385.166436] LustreError: 151898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6386.392158] LustreError: 152669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116600 with bad export cookie 6739527268044907238 [ 6386.393461] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6386.413491] LustreError: 152669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6386.989148] Lustre: server umount lustre-MDT0001 complete [ 6391.335640] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 04:50:02 (1789116602) [ 6393.338826] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6395.538134] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 04:50:06 (1789116606) [ 6405.112071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.597407] LustreError: 155898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6405.698467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6410.347705] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6410.729906] LustreError: 155899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6415.858442] LustreError: 155898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6419.327183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6423.953761] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6430.036315] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6433.705619] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6441.314456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6441.947456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.965614] Lustre: Skipped 1 previous similar message [ 6447.973670] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6454.931960] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6465.507579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6465.511739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6465.533287] Lustre: Skipped 3 previous similar messages [ 6465.552344] Lustre: Skipped 3 previous similar messages [ 6467.411208] Lustre: server umount lustre-MDT0000 complete [ 6470.625265] LustreError: 155898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6470.652339] LustreError: 155898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6470.935772] LustreError: 155878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116684 with bad export cookie 6739527268044908120 [ 6470.948928] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6470.957375] LustreError: 155878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6475.749411] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.777030] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6475.796896] Lustre: Skipped 1 previous similar message [ 6477.392784] Lustre: server umount lustre-MDT0001 complete [ 6484.398629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6499.561780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6499.939453] LustreError: 159398:0:(llog.c:1646:llog_backup()) MGC192.168.203.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6499.972348] Lustre: 159398:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.203.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6505.378723] LustreError: 159403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6505.427311] LustreError: 159403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6521.824103] LustreError: 159420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6521.833621] LustreError: 159420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6556.681734] LustreError: 159405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6556.714741] LustreError: 159405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 6560.739586] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6560.742910] Lustre: 159398:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.149@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6601.724668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6606.131434] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6606.825064] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6613.755405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.066981] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5d879c57bde03f8e [ 6617.078905] Lustre: MGC192.168.203.149@tcp: Connection restored to 0@lo (at 0@lo) [ 6621.171364] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6622.704421] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6626.314927] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6627.775405] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6630.944549] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6632.594201] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6653.408490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6653.410214] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6653.418550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6653.418558] Lustre: Skipped 1 previous similar message [ 6653.465142] Lustre: Skipped 2 previous similar messages [ 6657.349512] Lustre: server umount lustre-MDT0000 complete [ 6658.529723] LustreError: 159403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6658.560248] LustreError: 159403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 44 previous similar messages [ 6660.606298] LustreError: 157514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116874 with bad export cookie 6739527268044914574 [ 6660.613838] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6660.627488] LustreError: 157514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6660.970769] Lustre: server umount lustre-MDT0001 complete [ 6674.571987] Lustre: server umount lustre-OST0000 complete [ 6698.101456] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 6708.715498] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6741.804074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6742.060876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6742.090495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6742.143845] Lustre: lustre-MDT0000: new disk, initializing [ 6742.222909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6742.238285] Lustre: Skipped 1 previous similar message [ 6742.299323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6746.563265] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6756.454151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6756.551791] Lustre: 164255:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6756.763116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6761.160478] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6769.255337] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6773.096702] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6780.256581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6780.492161] Lustre: lustre-OST0000: new disk, initializing [ 6780.496677] Lustre: Skipped 1 previous similar message [ 6780.516479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6780.524062] Lustre: Skipped 2 previous similar messages [ 6780.529594] Lustre: 165998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6787.690163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6787.700912] Lustre: Skipped 1 previous similar message [ 6787.704787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6787.854353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6788.028364] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6797.533164] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6802.957546] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6803.264455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6808.022269] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6808.297717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6813.670514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6813.679186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.708671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6813.719027] Lustre: Skipped 2 previous similar messages [ 6816.643456] Lustre: server umount lustre-OST0000 complete [ 6823.393095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6823.400384] LustreError: Skipped 1 previous similar message [ 6835.167203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6835.484520] Lustre: server umount lustre-MDT0000 complete [ 6838.240039] LustreError: 164263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.260194] LustreError: 164263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6839.716572] LustreError: 165997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117053 with bad export cookie 6739527268044931409 [ 6839.719415] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6839.731283] LustreError: 165997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6840.077211] Lustre: server umount lustre-MDT0001 complete [ 6848.917372] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 04:57:40 (1789117060) [ 6855.745962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.869646] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6871.336179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6875.559803] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6880.529342] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6884.493837] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6890.479678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6896.501509] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6902.272212] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6912.806708] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 04:58:43 (1789117123) [ 6917.600733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.608509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6917.621351] Lustre: Skipped 3 previous similar messages [ 6917.626401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6917.646758] Lustre: Skipped 6 previous similar messages [ 6923.830653] Lustre: server umount lustre-OST0000 complete [ 6935.704821] Lustre: server umount lustre-MDT0000 complete [ 6940.466573] LustreError: 168985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117154 with bad export cookie 6739527268044932291 [ 6940.476544] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6940.495958] LustreError: 168985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6941.008789] Lustre: server umount lustre-MDT0001 complete [ 6954.805844] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6957.563981] Key type lgssc unregistered [ 6957.864498] LNet: 171635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6957.868463] LNetError: 171635:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6957.895508] LNet: Removed LNI 192.168.203.149@tcp [ 6958.874710] Key type .llcrypt unregistered [ 6958.876633] Key type ._llcrypt unregistered [ 6978.415598] Key type ._llcrypt registered [ 6978.421730] Key type .llcrypt registered [ 6978.558905] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6979.670148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6979.722748] alg: No test for adler32 (adler32-zlib) [ 6980.794909] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 6981.052946] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6982.727224] Key type lgssc registered [ 6983.735429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6986.229362] alg: No test for zstdfast (zstdfast-lustre-generic) [ 6986.236877] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 6992.056129] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6992.090491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7007.712215] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7012.833159] LustreError: 172948:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 7017.952203] LustreError: 172948:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7017.967761] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7017.978200] LustreError: MGC192.168.203.149@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7017.992700] LustreError: 172948:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7018.002661] LustreError: 172948:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7018.008805] LustreError: 172948:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7018.114540] Lustre: server umount lustre-OST0000 complete [ 7018.123357] LustreError: 172948:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7026.629447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7028.073833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7032.405890] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7037.429240] LustreError: 173448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7037.444819] LustreError: 173448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7041.391295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7041.688056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7046.419276] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7052.614134] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7056.482579] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7063.232148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7063.414300] LustreError: 175047:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7063.424260] Lustre: 175047:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.203.149@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7063.594665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.646885] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7074.843471] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7077.356856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7077.367675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.387475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7077.878697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.899459] Lustre: Skipped 1 previous similar message [ 7077.903892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7077.917213] Lustre: Skipped 1 previous similar message [ 7082.993828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7083.008832] Lustre: Skipped 2 previous similar messages [ 7083.606632] Lustre: server umount lustre-MDT0000 complete [ 7088.097492] LustreError: 174568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.116148] LustreError: 174568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7088.374254] LustreError: 173428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117302 with bad export cookie 2001644771467831062 [ 7088.389131] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7088.662727] Lustre: server umount lustre-MDT0001 complete [ 7101.369690] Lustre: server umount lustre-OST0000 complete [ 7109.530493] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 7117.527995] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 7148.739904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7148.943426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7148.962733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7149.019400] Lustre: lustre-MDT0000: new disk, initializing [ 7149.111969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7149.125236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7153.210163] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7163.117234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.248813] Lustre: 178438:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7163.336484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7163.340967] Lustre: Skipped 1 previous similar message [ 7163.463694] Lustre: lustre-MDT0001: new disk, initializing [ 7163.591091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7163.640704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7163.652541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7168.848759] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7176.064192] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7180.329209] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7188.603142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7189.076700] Lustre: lustre-OST0000: new disk, initializing [ 7189.083955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7189.090556] Lustre: 180179:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7189.229040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7197.030463] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7198.805157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7198.810539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7198.940795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7205.168903] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7209.780679] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7210.040837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7214.536426] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7214.781129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7219.679853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7219.685516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.704940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7219.710413] Lustre: Skipped 1 previous similar message [ 7219.723830] Lustre: Skipped 1 previous similar message [ 7222.711582] Lustre: server umount lustre-OST0000 complete [ 7229.920597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.957353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7229.974751] Lustre: Skipped 1 previous similar message [ 7239.139761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7239.147400] Lustre: Skipped 3 previous similar messages [ 7242.207324] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7242.567343] Lustre: server umount lustre-MDT0000 complete [ 7244.258222] LustreError: 178448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7244.283756] LustreError: 178448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7246.182186] LustreError: 178429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117460 with bad export cookie 2001644771467831783 [ 7246.185750] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7246.193247] LustreError: 178429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7246.424455] Lustre: server umount lustre-MDT0001 complete [ 7254.293484] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 05:04:26 (1789117466) [ 7269.541270] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 7272.978631] Key type lgssc unregistered [ 7273.371457] LNet: 182962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7273.403197] LNetError: 182962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7273.460073] LNet: Removed LNI 192.168.203.149@tcp [ 7274.765946] Key type .llcrypt unregistered [ 7274.768512] Key type ._llcrypt unregistered [ 7298.224288] Key type ._llcrypt registered [ 7298.226190] Key type .llcrypt registered [ 7298.323929] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 7299.174571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7299.398983] alg: No test for adler32 (adler32-zlib) [ 7300.677170] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 7301.035647] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 7302.687278] Key type lgssc registered [ 7304.408145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7308.122855] alg: No test for zstdfast (zstdfast-lustre-generic) [ 7308.146238] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 7313.594444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7313.610458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.029234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7319.317069] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7324.133020] LustreError: 184249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7324.152638] LustreError: 184249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7327.570162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.035058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7332.305048] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7337.668039] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7341.746612] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7351.176501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7351.338444] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7351.340755] LustreError: 185889:0:(llog.c:1687:llog_backup()) MGC192.168.203.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7351.347718] Lustre: 185889:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.203.149@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7351.504091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7358.085561] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7363.567510] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7369.185565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7369.193713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7369.204678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7370.209291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7370.212574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7370.224465] Lustre: Skipped 1 previous similar message [ 7370.254311] Lustre: Skipped 1 previous similar message [ 7371.601105] Lustre: server umount lustre-MDT0000 complete [ 7374.919064] LustreError: 186538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117588 with bad export cookie 13167833270397837126 [ 7374.930865] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7374.934469] LustreError: 186538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7375.328137] LustreError: 184245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7375.328502] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7375.473154] Lustre: server umount lustre-MDT0001 complete [ 7381.473455] Lustre: server umount lustre-OST0000 complete [ 7390.362796] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 7399.793798] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 7435.372586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7435.626563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7435.666928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7435.775276] Lustre: lustre-MDT0000: new disk, initializing [ 7435.933209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7435.970297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7439.854297] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7449.179670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.292661] Lustre: 189283:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7449.338400] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7449.341219] Lustre: Skipped 1 previous similar message [ 7449.344960] LustreError: 190046:0:(llog.c:1687:llog_backup()) MGC192.168.203.149@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7449.353020] LustreError: 190046:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7449.359604] Lustre: 190046:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.203.149@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7449.375791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7449.380116] Lustre: Skipped 1 previous similar message [ 7449.441899] Lustre: lustre-MDT0001: new disk, initializing [ 7449.526212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7449.578513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7449.597648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7453.181139] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7459.506428] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7462.579433] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7467.600659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7467.815403] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7467.818593] Lustre: Skipped 1 previous similar message [ 7467.822325] LustreError: 190997:0:(llog.c:1687:llog_backup()) MGC192.168.203.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7467.828841] LustreError: 190997:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7467.833494] Lustre: 190997:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.203.149@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7467.842369] Lustre: 190997:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7467.862530] Lustre: lustre-OST0000: new disk, initializing [ 7467.867566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7467.874517] Lustre: 191022:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7467.944386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7473.914345] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7474.771787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7474.784579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7474.894604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7482.427568] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7487.789561] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7488.133456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7492.461485] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7492.654885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7495.660705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7495.662302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7495.668157] LustreError: Skipped 1 previous similar message [ 7495.686285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7495.693172] Lustre: Skipped 2 previous similar messages [ 7500.779126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7500.790731] Lustre: Skipped 1 previous similar message [ 7500.980888] Lustre: server umount lustre-OST0000 complete [ 7505.888291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7505.914254] Lustre: Skipped 1 previous similar message [ 7505.930035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7505.939019] Lustre: Skipped 1 previous similar message [ 7510.837162] Lustre: server umount lustre-MDT0000 complete [ 7511.013923] LustreError: 189293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7511.020352] LustreError: 189293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7514.603108] LustreError: 189273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117728 with bad export cookie 13167833270397837847 [ 7514.604436] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7514.612794] LustreError: 189273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7514.869208] Lustre: server umount lustre-MDT0001 complete [ 7523.222451] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 05:08:54 (1789117734) [ 7539.627775] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 7550.368881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7550.843268] LustreError: 193997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.912681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7555.528838] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7556.065287] LustreError: 193998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7564.716902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7569.742983] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7573.020769] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7579.894824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7587.034974] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7596.302389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7596.485239] Lustre: lustre-OST0001: new disk, initializing [ 7596.488437] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7596.495264] Lustre: 196253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7596.615487] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7596.623549] Lustre: Skipped 2 previous similar messages [ 7602.767133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7602.782648] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7602.905514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7603.366575] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7616.205911] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7628.639961] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7630.907278] Lustre: DEBUG MARKER: ADD OST3 [ 7636.441968] loop: module loaded [ 7636.597349] loop0: detected capacity change from 0 to 204800000 [ 7636.938995] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7636.956819] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7636.980303] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7644.319179] loop0: detected capacity change from 0 to 204800000 [ 7644.323656] Lustre: DEBUG MARKER: STOP OST3 [ 7651.276313] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7651.544345] Lustre: lustre-OST0003: new disk, initializing [ 7651.547904] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7651.563800] Lustre: 198286:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7653.147713] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7653.327768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7653.356419] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7653.422076] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7654.923452] Lustre: Failing over lustre-OST0003 [ 7655.201929] LustreError: 198407:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7655.205401] Lustre: server umount lustre-OST0003 complete [ 7658.466087] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7658.480825] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7658.512104] Lustre: Skipped 1 previous similar message [ 7658.529061] LustreError: 195448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.576081] LustreError: 195448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7659.496563] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7667.861222] Lustre: DEBUG MARKER: START OST3 again [ 7675.126374] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7675.655243] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7675.664014] Lustre: Skipped 1 previous similar message [ 7675.692073] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7676.917553] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7677.528720] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7677.536324] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7682.439629] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7704.688100] Lustre: Failing over lustre-OST0003 [ 7704.786173] Lustre: server umount lustre-OST0003 complete [ 7706.601591] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7706.615966] LustreError: 195449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7706.618867] Lustre: Skipped 2 previous similar messages [ 7706.642431] LustreError: 195449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 7724.004598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7724.011955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7728.889421] Lustre: server umount lustre-MDT0000 complete [ 7732.652584] LustreError: 193978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117946 with bad export cookie 13167833270397838729 [ 7732.672341] LustreError: 193978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7732.672492] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7739.148249] Lustre: server umount lustre-MDT0001 complete [ 7743.278811] Lustre: server umount lustre-OST0000 complete [ 7747.339972] Lustre: server umount lustre-OST0001 complete [ 7753.124503] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 7760.611832] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 7790.880892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.164052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7791.197116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7791.251237] Lustre: lustre-MDT0000: new disk, initializing [ 7791.337638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7795.153893] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7805.906775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.066274] Lustre: 203641:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7806.481787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7806.490168] Lustre: Skipped 1 previous similar message [ 7806.552332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7811.316792] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7819.898802] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7824.023787] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7832.683474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7832.982189] Lustre: lustre-OST0000: new disk, initializing [ 7832.997023] Lustre: Skipped 1 previous similar message [ 7833.002677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7833.006914] Lustre: Skipped 2 previous similar messages [ 7833.015397] Lustre: 205389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7834.563423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7834.567143] Lustre: Skipped 1 previous similar message [ 7834.569396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7834.615358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7838.215893] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7845.254991] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7849.559930] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7849.853019] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7854.002532] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7854.215990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7860.193138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7860.207364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7860.230604] Lustre: Skipped 6 previous similar messages [ 7860.242388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7860.255145] Lustre: Skipped 7 previous similar messages [ 7862.203499] Lustre: server umount lustre-OST0000 complete [ 7880.159491] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7880.444102] Lustre: server umount lustre-MDT0000 complete [ 7880.686508] LustreError: 203654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7880.709988] LustreError: 203654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 7884.288785] LustreError: 203636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118098 with bad export cookie 13167833270397841018 [ 7884.296287] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7884.305127] LustreError: 203636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7884.563935] Lustre: server umount lustre-MDT0001 complete [ 7888.314557] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 05:14:59 (1789118099) [ 7905.204689] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 7914.627571] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 7948.695655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.919264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7948.954397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7949.036373] Lustre: lustre-MDT0000: new disk, initializing [ 7949.131540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7953.579426] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7964.816520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.949576] Lustre: 209830:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7965.271977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7969.567330] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7976.886961] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7980.417710] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7986.983667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7987.301213] Lustre: 211575:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7989.494611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7993.431517] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8000.757729] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8005.711773] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8005.944080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8010.552597] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8010.781473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8014.817371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8014.829680] LustreError: Skipped 2 previous similar messages [ 8014.835433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.847064] Lustre: Skipped 3 previous similar messages [ 8014.854361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8014.860045] Lustre: Skipped 5 previous similar messages [ 8018.738865] Lustre: server umount lustre-OST0000 complete [ 8037.346081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8037.606885] Lustre: server umount lustre-MDT0000 complete [ 8040.935657] LustreError: 209841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8040.954991] LustreError: 209841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8041.834271] LustreError: 209823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118255 with bad export cookie 13167833270397841907 [ 8041.842505] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8041.848266] LustreError: 209823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8042.289886] Lustre: server umount lustre-MDT0001 complete [ 8050.269905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.375801] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8064.075702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8064.518820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8064.525733] Lustre: Skipped 5 previous similar messages [ 8068.256981] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8073.214316] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8076.728159] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8084.071113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8090.673736] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8097.456695] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8110.048013] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8110.051213] LustreError: Skipped 1 previous similar message [ 8110.057114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8110.060157] Lustre: Skipped 8 previous similar messages [ 8114.421528] Lustre: server umount lustre-OST0000 complete [ 8125.220029] Lustre: server umount lustre-MDT0000 complete [ 8129.346609] LustreError: 213624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118343 with bad export cookie 13167833270397842782 [ 8129.347793] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8129.359641] LustreError: 213624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8130.027605] Lustre: server umount lustre-MDT0001 complete [ 8238.641875] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 8241.661751] Key type lgssc unregistered [ 8241.978624] LNet: 216950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8241.990205] LNetError: 216950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8242.008635] LNet: Removed LNI 192.168.203.149@tcp [ 8242.830362] Key type .llcrypt unregistered [ 8242.832502] Key type ._llcrypt unregistered [ 8266.507648] Key type ._llcrypt registered [ 8266.510659] Key type .llcrypt registered [ 8266.612616] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 8281.516577] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 8282.500789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8282.614828] alg: No test for adler32 (adler32-zlib) [ 8283.594873] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 8283.799127] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 8285.471572] Key type lgssc registered [ 8286.568307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8288.882770] alg: No test for zstdfast (zstdfast-lustre-generic) [ 8288.893335] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 8320.274072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8320.305704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8321.755572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8321.832432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8321.950296] Lustre: lustre-MDT0000: new disk, initializing [ 8322.050898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8322.107048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8328.285372] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8340.380159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.492464] Lustre: 220351:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8340.549550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8340.554068] Lustre: Skipped 1 previous similar message [ 8340.620653] Lustre: lustre-MDT0001: new disk, initializing [ 8340.701881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8340.739074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8340.751347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8345.164620] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8352.672412] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8356.301178] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8362.769949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8362.964793] Lustre: lustre-OST0000: new disk, initializing [ 8362.968296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8362.973334] Lustre: 222092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8363.044334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8368.009677] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8369.789745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8369.806790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8369.897576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8374.898492] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8379.495293] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8379.744452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8383.693946] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8383.884905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8390.632614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8390.633517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8390.644828] LustreError: Skipped 1 previous similar message [ 8390.645950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8390.654421] Lustre: Skipped 1 previous similar message [ 8390.678151] Lustre: Skipped 1 previous similar message [ 8391.852743] Lustre: server umount lustre-OST0000 complete [ 8395.748034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8395.761348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8395.771396] Lustre: Skipped 1 previous similar message [ 8395.792728] Lustre: Skipped 1 previous similar message [ 8400.866636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8400.871728] Lustre: Skipped 1 previous similar message [ 8401.598206] Lustre: server umount lustre-MDT0000 complete [ 8404.713020] LustreError: 220345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118618 with bad export cookie 11779428572370834709 [ 8404.714100] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8404.727271] LustreError: 220345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8405.024866] Lustre: server umount lustre-MDT0001 complete [ 8421.113522] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 8431.370214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8431.827844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8435.980073] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8437.220618] LustreError: 224873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8437.247413] LustreError: 224873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8441.319949] LustreError: 224872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8444.877042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8445.316332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8449.537723] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8452.227059] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8457.993889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8458.281703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8463.711983] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8471.610257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8471.765497] Lustre: lustre-OST0001: new disk, initializing [ 8471.769708] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8471.776763] Lustre: 227128:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8477.464394] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8478.309448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8478.314858] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8478.417074] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8487.990695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8496.940511] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8511.854554] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 05:25:23 (1789118723) [ 8514.534467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.537087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8514.545460] Lustre: Skipped 1 previous similar message [ 8514.551912] Lustre: Skipped 3 previous similar messages [ 8519.653921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8519.659862] Lustre: Skipped 3 previous similar messages [ 8520.574168] Lustre: server umount lustre-MDT0000 complete [ 8524.486874] LustreError: 224855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118738 with bad export cookie 11779428572370835591 [ 8524.494824] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8524.502340] LustreError: 224855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8524.769971] LustreError: 224868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8524.794241] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8524.816394] Lustre: Skipped 2 previous similar messages [ 8525.064942] Lustre: server umount lustre-MDT0001 complete [ 8531.456211] Lustre: server umount lustre-OST0000 complete [ 8535.139633] Lustre: server umount lustre-OST0001 complete [ 8542.044404] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 8553.250061] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 8592.057634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8592.298481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8592.321797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8592.373422] Lustre: lustre-MDT0000: new disk, initializing [ 8592.452258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8592.459240] Lustre: Skipped 1 previous similar message [ 8592.474121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8596.530859] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8607.331409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8607.441222] Lustre: 231037:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8607.503793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8607.512514] Lustre: Skipped 1 previous similar message [ 8607.615241] Lustre: lustre-MDT0001: new disk, initializing [ 8607.785515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8607.801602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8612.034914] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8619.062743] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8623.218096] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8629.471036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8629.692392] Lustre: lustre-OST0000: new disk, initializing [ 8629.696285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8629.702631] Lustre: 232780:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8629.772322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8629.778035] Lustre: Skipped 1 previous similar message [ 8630.791519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8630.801718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8630.934501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8635.770948] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8642.963601] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8647.510627] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8647.789943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8652.386229] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8652.567521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8656.353482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8656.357347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8656.385643] Lustre: Skipped 1 previous similar message [ 8656.389651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8656.421811] LustreError: Skipped 1 previous similar message [ 8660.577301] Lustre: server umount lustre-OST0000 complete [ 8664.033891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8666.595730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.608111] Lustre: Skipped 2 previous similar messages [ 8676.833610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8676.839252] Lustre: Skipped 7 previous similar messages [ 8678.367857] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8678.625377] Lustre: server umount lustre-MDT0000 complete [ 8681.968645] LustreError: 231048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8681.988446] LustreError: 231048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8682.325549] LustreError: 231028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118896 with bad export cookie 11779428572370837719 [ 8682.334270] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8682.339830] LustreError: 231028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8682.697711] Lustre: server umount lustre-MDT0001 complete [ 8690.872397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8691.295383] LustreError: 234851:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8695.562680] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8701.924539] LustreError: 234852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8701.957444] LustreError: 234852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8703.270546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8703.686671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8703.695793] Lustre: Skipped 1 previous similar message [ 8708.160706] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8713.030509] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8716.683875] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8723.118294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8731.286862] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 8738.535829] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8885.218944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8885.223421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8885.226144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8885.243392] Lustre: Skipped 1 previous similar message [ 8890.347397] Lustre: server umount lustre-OST0000 complete [ 8899.535089] Lustre: server umount lustre-MDT0000 complete [ 8900.583124] LustreError: 234852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8900.606474] LustreError: 234852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8903.784594] LustreError: 234831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119117 with bad export cookie 11779428572370838608 [ 8903.786235] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8903.799552] LustreError: 234831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8904.155595] Lustre: server umount lustre-MDT0001 complete [ 9008.265636] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 9011.191225] Key type lgssc unregistered [ 9011.538661] LNet: 238103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9011.545487] LNetError: 238103:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9011.576793] LNet: Removed LNI 192.168.203.149@tcp [ 9012.551129] Key type .llcrypt unregistered [ 9012.553571] Key type ._llcrypt unregistered [ 9039.661367] Key type ._llcrypt registered [ 9039.665714] Key type .llcrypt registered [ 9039.846854] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 9058.403706] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 9059.370570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9059.402227] alg: No test for adler32 (adler32-zlib) [ 9060.432658] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [ 9060.634434] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 9062.279150] Key type lgssc registered [ 9063.382855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9065.962346] alg: No test for zstdfast (zstdfast-lustre-generic) [ 9065.974591] alg: No test for zstdfast (zstdfast-lustre-scomp) [ 9118.659553] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 9129.572175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9129.596051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9130.858930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9130.880473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9131.000283] Lustre: lustre-MDT0000: new disk, initializing [ 9131.161515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9131.194756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9136.054935] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9146.152222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9146.306994] Lustre: 242222:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9146.425932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9146.429662] Lustre: Skipped 1 previous similar message [ 9146.503038] Lustre: lustre-MDT0001: new disk, initializing [ 9146.595743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9146.647092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9146.659884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9151.692647] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9156.662455] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9164.737113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9165.134819] Lustre: lustre-OST0000: new disk, initializing [ 9165.143414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9165.161620] Lustre: 243818:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9165.311765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9166.257808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9166.276313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9166.377389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9173.024605] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9183.080899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9183.256244] Lustre: lustre-OST0001: new disk, initializing [ 9183.259173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9183.266904] Lustre: 244673:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9183.374806] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9188.638548] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9191.590160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9191.603853] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9191.714565] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9199.638265] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9210.509577] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9222.627447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9222.633367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9222.652655] Lustre: Skipped 3 previous similar messages [ 9222.670793] Lustre: Skipped 3 previous similar messages [ 9223.504912] Lustre: server umount lustre-MDT0000 complete [ 9226.436259] LustreError: 243817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119440 with bad export cookie 6693141799292636871 [ 9226.441224] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9226.445469] LustreError: 243817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9226.741977] Lustre: server umount lustre-MDT0001 complete [ 9241.240759] Lustre: server umount lustre-OST0000 complete [ 9244.128061] Lustre: 239680:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789119441/real 1789119441] req@ffff8e5609939f80 x1876027531930368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789119457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9244.174142] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9245.274529] Lustre: server umount lustre-OST0001 complete [ 9257.872609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9267.476787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9275.686965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9284.564162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9305.227664] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 9315.743795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9315.787594] Lustre: lustre-MDT0000: reset Object Index mappings [ 9316.541867] LustreError: 249663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9316.561482] LustreError: 249663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9316.674594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9321.833862] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9321.957712] LustreError: 249664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.081102] LustreError: 249663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9332.208571] LustreError: 249664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9332.427695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9332.456364] Lustre: lustre-MDT0001: reset Object Index mappings [ 9333.171875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9339.302733] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9342.911905] Lustre: 250772:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9350.776392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9351.434600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9357.549926] LustreError: 251125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9358.551369] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9361.675672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9367.014062] LustreError: 251127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9367.036354] LustreError: 251127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9368.059802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9373.706965] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9374.396525] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9382.418572] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9386.034430] Lustre: 252610:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9392.945109] Lustre: server umount lustre-MDT0000 complete [ 9394.150228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.160595] LustreError: 249658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9394.170045] Lustre: Skipped 3 previous similar messages [ 9394.204749] LustreError: 249658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9396.436771] LustreError: 249643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119610 with bad export cookie 6693141799292643661 [ 9396.440337] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9396.457716] LustreError: 249643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9396.807848] Lustre: server umount lustre-MDT0001 complete [ 9411.148992] Lustre: server umount lustre-OST0000 complete [ 9415.455144] Lustre: 239677:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789119613/real 1789119613] req@ffff8e5608b49880 x1876027531981056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789119629 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9415.483577] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9416.004121] Lustre: server umount lustre-OST0001 complete [ 9436.822558] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 05:40:48 (1789119648) [ 9451.724374] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 9462.054501] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 9496.421472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9496.695900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9496.717802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9496.784869] Lustre: lustre-MDT0000: new disk, initializing [ 9496.900435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.903451] Lustre: Skipped 1 previous similar message [ 9496.924964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9500.993948] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9510.136868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9510.276930] Lustre: 256843:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9510.370954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9510.376066] Lustre: Skipped 1 previous similar message [ 9510.462845] Lustre: lustre-MDT0001: new disk, initializing [ 9510.575763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9510.591504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9514.655859] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9521.493433] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9525.498896] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9532.223676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9532.440931] Lustre: lustre-OST0000: new disk, initializing [ 9532.444360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9532.450756] Lustre: 258584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9533.674270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9533.692867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9533.880189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9539.224991] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9546.373258] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9551.002904] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9551.230059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9555.681835] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9555.906432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9559.522419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9559.529381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9559.530832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9559.532588] LustreError: Skipped 1 previous similar message [ 9559.578767] Lustre: Skipped 1 previous similar message [ 9563.875967] Lustre: server umount lustre-OST0000 complete [ 9569.763290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9569.791292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9569.802689] Lustre: Skipped 2 previous similar messages [ 9573.943071] Lustre: server umount lustre-MDT0000 complete [ 9574.887096] LustreError: 256856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9574.904218] LustreError: 256856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9577.364772] LustreError: 256836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119791 with bad export cookie 6693141799292645754 [ 9577.382540] LustreError: 256836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9577.395484] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9577.683798] Lustre: server umount lustre-MDT0001 complete [ 9597.386456] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 9607.801668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9608.419667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9608.426933] Lustre: Skipped 2 previous similar messages [ 9613.213380] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9622.412391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9627.599333] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9630.629804] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9638.624756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9646.409686] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9657.444243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9657.772779] Lustre: lustre-OST0001: new disk, initializing [ 9657.775737] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9657.792938] Lustre: 263626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9659.770694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9659.787637] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9659.921677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9664.451606] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9673.767149] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9677.805427] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9683.223305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9683.304304] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9694.690214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9694.694982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9694.717424] Lustre: Skipped 1 previous similar message [ 9694.723345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9699.275696] Lustre: server umount lustre-MDT0000 complete [ 9700.842327] LustreError: 261362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9700.864422] LustreError: 261362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9703.086717] LustreError: 264370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119916 with bad export cookie 6693141799292646636 [ 9703.093681] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9703.094775] LustreError: 264370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9703.545454] Lustre: server umount lustre-MDT0001 complete [ 9717.318126] Lustre: server umount lustre-OST0000 complete [ 9721.311421] Lustre: 239679:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789119919/real 1789119919] req@ffff8e5608b12d80 x1876027532085504/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789119935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9721.771691] Lustre: server umount lustre-OST0001 complete [ 9741.323171] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 9752.335656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9752.894153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9752.903176] Lustre: Skipped 3 previous similar messages [ 9757.635363] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9766.935670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9772.076918] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9775.188195] Lustre: 267760:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9782.375244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9789.113433] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9797.255652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9798.298527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9803.776353] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9805.812600] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9813.684001] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9817.403404] Lustre: 269603:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9822.695675] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9822.820708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9832.071415] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 05:47:23 (1789120043) [ 9849.314078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9849.323551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9849.332280] Lustre: Skipped 4 previous similar messages [ 9849.338431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9849.344941] Lustre: Skipped 4 previous similar messages [ 9853.666856] Lustre: server umount lustre-MDT0000 complete [ 9854.945947] LustreError: 266655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9854.991847] LustreError: 266655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 9857.222541] LustreError: 266638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120071 with bad export cookie 6693141799292652978 [ 9857.229950] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9857.238814] LustreError: 266638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9860.071555] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9860.083341] Lustre: Skipped 4 previous similar messages [ 9863.628304] Lustre: server umount lustre-MDT0001 complete [ 9873.641102] Lustre: server umount lustre-OST0000 complete [ 9877.251530] Lustre: server umount lustre-OST0001 complete [ 9883.855439] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 9892.944874] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 9940.815704] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 9952.696699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9953.107144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9953.150588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9953.217177] Lustre: lustre-MDT0000: new disk, initializing [ 9953.374253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9958.131924] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9969.037724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9969.133205] Lustre: 273822:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9969.520497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9974.199031] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 9987.501472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9988.011222] Lustre: 275424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9990.077221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9997.347073] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10011.052638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10011.208370] Lustre: 276279:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10011.311495] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10011.323308] Lustre: Skipped 6 previous similar messages [10012.733381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10012.738306] Lustre: Skipped 1 previous similar message [10012.837136] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10018.185228] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10028.358882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10032.125191] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10032.130177] Lustre: Skipped 1 previous similar message [10046.365882] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 05:50:57 (1789120257) [10048.995770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.001498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10049.010059] Lustre: Skipped 6 previous similar messages [10049.024115] Lustre: Skipped 5 previous similar messages [10050.520692] Lustre: server umount lustre-MDT0000 complete [10053.554895] LustreError: 275050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120267 with bad export cookie 6693141799292660846 [10053.560285] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10053.563106] LustreError: 275050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10053.901502] Lustre: server umount lustre-MDT0001 complete [10066.424031] Lustre: server umount lustre-OST0000 complete [10069.348417] Lustre: 239679:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789120267/real 1789120267] req@ffff8e56076e1180 x1876027532234752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789120283 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10069.879289] Lustre: server umount lustre-OST0001 complete [10077.854862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10081.472417] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10089.461244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10092.941286] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10098.309916] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10102.312404] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10109.584586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10114.152394] LustreError: 280112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.179654] LustreError: 280112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10117.041274] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10123.399677] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10139.856267] loop1: detected capacity change from 0 to 204800000 [10140.033223] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10140.048619] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10140.064341] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10144.382736] loop1: detected capacity change from 0 to 204800000 [10148.962306] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10149.046033] LustreError: 278497:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs27729: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10149.063758] LustreError: 278497:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs27729 [10149.072465] LustreError: 278497:0:(mgs_handler.c:517:mgs_target_reg()) fs27729-MDT0001: Error checking target index: rc = -13 [10149.085253] LustreError: fs27729-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10149.090947] LustreError: 281934:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10149.096811] LustreError: 281934:0:(tgt_mount.c:2084:server_put_super()) no obd fs27729-MDT0001 [10149.101672] LustreError: 281934:0:(tgt_mount.c:116:server_deregister_mount()) fs27729-MDT0001 not registered [10149.166953] Lustre: server umount fs27729-MDT0001 complete [10149.170050] LustreError: 281934:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10155.824588] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10155.926789] Lustre: Setting parameter fs27729-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs27729-MDT0001 [10155.979599] Lustre: srv-fs27729-MDT0001: No data found on store. Initialize space. [10155.984477] Lustre: Skipped 4 previous similar messages [10156.061674] Lustre: fs27729-MDT0001: new disk, initializing [10156.064679] Lustre: Skipped 3 previous similar messages [10156.155973] Lustre: srv-fs27729-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10156.165691] Lustre: 282400:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27729-MDT0001-mdtlov: can't create new llog: rc = -115 [10156.178807] LustreError: 282400:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27729-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10159.838691] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10165.208991] Lustre: server umount fs27729-MDT0001 complete [10175.898597] loop1: detected capacity change from 0 to 204800000 [10180.415560] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.781606] Lustre: 283952:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27729-MDT0001-mdtlov: can't create new llog: rc = -115 [10180.788118] LustreError: 283952:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27729-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10184.743340] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10189.223778] Lustre: server umount fs27729-MDT0001 complete [10200.545385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10200.568152] Lustre: Skipped 5 previous similar messages [10200.575349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10200.580405] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10200.589655] Lustre: Skipped 2 previous similar messages [10212.319742] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10212.486329] Lustre: server umount lustre-MDT0000 complete [10219.731927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10219.876377] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10223.615737] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10225.134308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10230.944471] loop1: detected capacity change from 0 to 204800000 [10236.086100] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.451625] Lustre: 286365:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27729-MDT0001-mdtlov: can't create new llog: rc = -115 [10236.465831] LustreError: 286365:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27729-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10241.155572] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10246.332978] Lustre: server umount fs27729-MDT0001 complete [10271.199995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10271.221278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10271.230460] Lustre: Skipped 10 previous similar messages [10275.775389] Lustre: server umount lustre-OST0000 complete [10281.441609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10294.239424] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10294.471644] Lustre: server umount lustre-MDT0000 complete [10298.292127] LustreError: 278490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120512 with bad export cookie 6693141799292663527 [10298.308924] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10298.328935] LustreError: 278490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10298.612358] Lustre: server umount lustre-MDT0001 complete [10310.888686] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [10314.272297] Key type lgssc unregistered [10314.593790] LNet: 288675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10314.604919] LNetError: 288675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10314.622775] LNet: Removed LNI 192.168.203.149@tcp [10315.490159] Key type .llcrypt unregistered [10315.492385] Key type ._llcrypt unregistered [10337.045150] Key type ._llcrypt registered [10337.048365] Key type .llcrypt registered [10337.155454] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [10338.360643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10338.481701] alg: No test for adler32 (adler32-zlib) [10339.601475] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [10339.825065] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [10341.479248] Key type lgssc registered [10342.720714] Lustre: Echo OBD driver; http://www.lustre.org/ [10345.934081] alg: No test for zstdfast (zstdfast-lustre-generic) [10345.956478] alg: No test for zstdfast (zstdfast-lustre-scomp) [10351.873209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10351.903365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.625889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10357.901700] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10363.875488] LustreError: 290015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10363.886394] LustreError: 290015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10366.927808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10367.292414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10371.198697] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10376.600954] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10380.724449] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10387.284393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10387.696320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10392.813800] LustreError: 291616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10393.961907] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10398.176863] LustreError: 291618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10399.804849] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10402.425363] LustreError: 291618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10402.467897] LustreError: 291618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10407.431556] LustreError: 291897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10407.467780] LustreError: 291897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10417.671140] LustreError: 291897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10417.703228] LustreError: 291897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10421.322685] loop1: detected capacity change from 0 to 204800000 [10421.620223] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10421.638195] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10421.664233] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10427.998825] loop1: detected capacity change from 0 to 204800000 [10434.084305] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10434.273997] Lustre: fs15818-OST0000: new disk, initializing [10434.281265] Lustre: srv-fs15818-OST0000: No data found on store. Initialize space. [10434.293838] Lustre: 293456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10434.405921] Lustre: fs15818-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10435.060240] LustreError: 291617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.091575] LustreError: 291617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [10440.941178] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10451.205514] Lustre: server umount fs15818-OST0000 complete [10465.438583] loop1: detected capacity change from 0 to 204800000 [10468.872587] LustreError: 291616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10468.897613] LustreError: 291616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [10471.687313] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10472.035361] Lustre: fs15818-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10476.022613] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10478.317533] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10481.157288] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10487.759631] loop2: detected capacity change from 0 to 204800000 [10487.981810] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10488.001249] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10488.016701] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10493.692923] loop2: detected capacity change from 0 to 204800000 [10500.329105] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10500.450063] LustreError: fs15818-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10500.470681] LustreError: fs15818-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10500.485029] LustreError: 296466:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10500.491890] LustreError: 296466:0:(tgt_mount.c:2084:server_put_super()) no obd fs15818-OST0001 [10500.497609] LustreError: 296466:0:(tgt_mount.c:116:server_deregister_mount()) fs15818-OST0001 not registered [10500.535614] Lustre: server umount fs15818-OST0001 complete [10500.539459] LustreError: 296466:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10511.613443] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10511.831831] Lustre: fs15818-OST0001: new disk, initializing [10511.836188] Lustre: srv-fs15818-OST0001: No data found on store. Initialize space. [10511.848671] Lustre: 297018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10511.975134] Lustre: fs15818-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10518.840791] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10526.688387] Lustre: server umount fs15818-OST0000 complete [10535.442668] LustreError: 291616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10535.483778] LustreError: 291616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [10536.475283] Lustre: server umount fs15818-OST0001 complete [10549.786390] loop1: detected capacity change from 0 to 204800000 [10557.257791] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10557.527961] Lustre: fs15818-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10565.356632] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10573.367752] loop2: detected capacity change from 0 to 204800000 [10579.201953] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10585.764572] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10594.840280] Lustre: server umount fs15818-OST0000 complete [10604.899376] Lustre: server umount fs15818-OST0001 complete [10619.364699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10619.385850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10619.409187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10622.435067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10622.444843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10625.403184] Lustre: server umount lustre-OST0000 complete [10632.677641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10632.684497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10632.701732] Lustre: Skipped 1 previous similar message [10632.708351] Lustre: Skipped 1 previous similar message [10638.598193] Lustre: server umount lustre-MDT0000 complete [10642.421671] LustreError: 297017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120856 with bad export cookie 2941558469651898952 [10642.429631] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10642.437879] LustreError: 297017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10643.025608] Lustre: server umount lustre-MDT0001 complete [10654.800476] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [10657.747200] Key type lgssc unregistered [10658.060870] LNet: 302884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10658.074506] LNetError: 302884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10658.095268] LNet: Removed LNI 192.168.203.149@tcp [10658.805580] Key type .llcrypt unregistered [10658.809352] Key type ._llcrypt unregistered [10690.667692] Key type ._llcrypt registered [10690.670657] Key type .llcrypt registered [10690.830109] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [10707.483616] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [10708.512317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10708.714901] alg: No test for adler32 (adler32-zlib) [10709.800181] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [10710.030065] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [10711.719194] Key type lgssc registered [10712.772933] Lustre: Echo OBD driver; http://www.lustre.org/ [10715.310917] alg: No test for zstdfast (zstdfast-lustre-generic) [10715.336472] alg: No test for zstdfast (zstdfast-lustre-scomp) [10749.119927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10749.154830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10750.362202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10750.408966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10750.488794] Lustre: lustre-MDT0000: new disk, initializing [10750.577674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10750.593684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10755.107767] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10765.498731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10765.691404] Lustre: 306797:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10765.768439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10765.775447] Lustre: Skipped 1 previous similar message [10765.843802] Lustre: lustre-MDT0001: new disk, initializing [10765.932965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10765.994095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10766.023923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10771.263836] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10779.033250] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10782.949724] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10790.839472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10791.206744] Lustre: lustre-OST0000: new disk, initializing [10791.215288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10791.222676] Lustre: 308536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10791.327724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10797.637966] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [10800.232916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10800.245569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10800.382633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10805.690851] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10808.125103] Lustre: Mounted lustre-client [10808.964405] LustreError: 309288:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8e560c167800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [10809.029044] Lustre: Unmounted lustre-client [10818.540989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10818.545393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10818.566740] Lustre: Skipped 1 previous similar message [10818.584809] Lustre: Skipped 1 previous similar message [10823.649913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10824.012531] Lustre: server umount lustre-OST0000 complete [10828.770631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10828.774878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10828.791735] Lustre: Skipped 1 previous similar message [10828.812212] Lustre: Skipped 2 previous similar messages [10834.008836] Lustre: server umount lustre-MDT0000 complete [10837.650274] LustreError: 306789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121051 with bad export cookie 2365713075101826427 [10837.654875] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10837.667909] LustreError: 306789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10838.045472] Lustre: server umount lustre-MDT0001 complete [10848.946752] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [10851.585454] Key type lgssc unregistered [10866.721266] LNet: 1 local NIs in recovery (showing 1): 192.168.203.149@tcp [10867.234389] LNet: 310571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10867.245711] LNetError: 310571:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10937.959047] LNet: Removed LNI 192.168.203.149@tcp [10938.822456] Key type .llcrypt unregistered [10938.825196] Key type ._llcrypt unregistered [10960.231732] Key type ._llcrypt registered [10960.234732] Key type .llcrypt registered [10960.338716] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [10979.196957] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [10980.122684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10980.144049] alg: No test for adler32 (adler32-zlib) [10981.247050] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [10981.538157] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [10983.231620] Key type lgssc registered [10984.661056] Lustre: Echo OBD driver; http://www.lustre.org/ [10987.967569] alg: No test for zstdfast (zstdfast-lustre-generic) [10987.977655] alg: No test for zstdfast (zstdfast-lustre-scomp) [11018.315957] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 06:07:09 (1789121229) [11026.264550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11026.298888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11027.579087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11027.603089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11027.648351] Lustre: lustre-MDT0000: new disk, initializing [11027.771791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11027.815917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11032.447897] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11043.575740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11043.672564] Lustre: 313846:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11043.725159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11043.727933] Lustre: Skipped 1 previous similar message [11043.778874] Lustre: lustre-MDT0001: new disk, initializing [11043.864765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11043.907321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11043.915155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11048.147214] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11056.526113] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11060.637085] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11067.278958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11067.581075] Lustre: lustre-OST0000: new disk, initializing [11067.584460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11067.591336] Lustre: 315587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11067.677133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11073.623319] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11075.151290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11075.160905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11075.321315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11081.793996] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11096.042431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11096.049077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11096.066833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11101.153499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11101.157883] Lustre: Skipped 1 previous similar message [11106.272314] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11106.284460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11106.301600] Lustre: Skipped 1 previous similar message [11106.447464] Lustre: server umount lustre-OST0000 complete [11110.371673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11110.385576] LustreError: Skipped 1 previous similar message [11110.390544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11110.410254] Lustre: Skipped 1 previous similar message [11110.416057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11110.419012] Lustre: Skipped 1 previous similar message [11111.396042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11116.553619] Lustre: server umount lustre-MDT0000 complete [11120.794873] LustreError: 313837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121334 with bad export cookie 17415501510004655682 [11120.806819] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11120.811858] LustreError: 313837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11121.148124] Lustre: server umount lustre-MDT0001 complete [11131.570309] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [11134.480697] Key type lgssc unregistered [11134.797455] LNet: 317218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11134.811226] LNetError: 317218:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11135.846361] LNet: Removed LNI 192.168.203.149@tcp [11136.838687] Key type .llcrypt unregistered [11136.846804] Key type ._llcrypt unregistered [11165.610463] Key type ._llcrypt registered [11165.614416] Key type .llcrypt registered [11165.704099] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [11166.687830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11166.756943] alg: No test for adler32 (adler32-zlib) [11167.833327] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [11168.064279] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [11169.719315] Key type lgssc registered [11170.983049] Lustre: Echo OBD driver; http://www.lustre.org/ [11174.236375] alg: No test for zstdfast (zstdfast-lustre-generic) [11174.248348] alg: No test for zstdfast (zstdfast-lustre-scomp) [11180.049660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11180.063298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11181.630668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11185.641367] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11191.778828] LustreError: 318706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.788943] LustreError: 318706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11194.256411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.718910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11199.324474] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11205.132712] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11208.910633] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11215.862404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11216.352243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11222.991855] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11228.621571] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11236.049548] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11237.780781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11242.980561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11242.984077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11242.995089] Lustre: Skipped 2 previous similar messages [11243.011130] Lustre: Skipped 2 previous similar messages [11245.613288] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.49@tcp (stopping) [11248.105944] LustreError: 318707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11248.127619] LustreError: 318707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11248.226372] Lustre: server umount lustre-MDT0000 complete [11250.698329] LustreError: 319914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11251.521661] LustreError: 318686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121465 with bad export cookie 17134943501211771465 [11251.537656] LustreError: 318686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11251.539781] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11253.217379] LustreError: 318702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11253.218471] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11253.278527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11255.843961] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.49@tcp (stopping) [11257.981850] Lustre: server umount lustre-MDT0001 complete [11262.333913] Lustre: server umount lustre-OST0000 complete [11272.043665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11277.080471] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11279.893972] Lustre: 322433:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.149@tcp [11281.905840] Lustre: server umount MGS complete [11291.194964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11291.814076] LustreError: 323004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.830497] LustreError: 323004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11291.870323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11295.437501] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11301.344970] LustreError: 323004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11301.371588] LustreError: 323004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11303.784393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11304.166292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11309.033719] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11317.406332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11317.789370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11324.000258] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11329.374103] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11331.723409] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11335.077358] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11338.939333] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [11344.352046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11344.360705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11344.371152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11344.374275] Lustre: Skipped 1 previous similar message [11346.401537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11349.485566] Lustre: server umount lustre-OST0000 complete [11355.616160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11355.626060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11355.644211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11355.653862] Lustre: Skipped 2 previous similar messages [11367.391521] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11367.728476] Lustre: server umount lustre-MDT0000 complete [11371.739392] LustreError: 322985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121585 with bad export cookie 17134943501211772305 [11371.745181] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11371.752440] LustreError: 322985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11372.052261] Lustre: server umount lustre-MDT0001 complete [11384.644192] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [11387.971290] Key type lgssc unregistered [11388.435051] LNet: 326182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11388.449474] LNetError: 326182:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11388.481926] LNet: Removed LNI 192.168.203.149@tcp [11389.363212] Key type .llcrypt unregistered [11389.365404] Key type ._llcrypt unregistered [11406.963413] Key type ._llcrypt registered [11406.969684] Key type .llcrypt registered [11407.049171] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [11422.011360] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [11423.102060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11423.163348] alg: No test for adler32 (adler32-zlib) [11424.208889] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [11424.429762] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [11426.135225] Key type lgssc registered [11427.364700] Lustre: Echo OBD driver; http://www.lustre.org/ [11430.214337] alg: No test for zstdfast (zstdfast-lustre-generic) [11430.224619] alg: No test for zstdfast (zstdfast-lustre-scomp) [11460.673705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11460.690033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11461.946437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11461.991274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11462.092708] Lustre: lustre-MDT0000: new disk, initializing [11462.188910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11462.208815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11466.217893] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11475.624343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11475.760250] Lustre: 329392:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11475.820641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11475.825826] Lustre: Skipped 1 previous similar message [11475.886169] Lustre: lustre-MDT0001: new disk, initializing [11475.956653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11475.985439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11475.993758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11480.114887] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11487.268344] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11490.231467] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11496.156028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11496.477540] Lustre: lustre-OST0000: new disk, initializing [11496.482334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11496.490521] Lustre: 331136:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11496.608858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11502.893700] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11504.721369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11504.738299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11504.821883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11509.512345] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11513.388305] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11513.573327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11517.645813] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11517.843403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11520.481437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11520.485698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11520.486668] LustreError: Skipped 1 previous similar message [11520.489039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11520.502312] Lustre: Skipped 1 previous similar message [11520.517048] Lustre: Skipped 1 previous similar message [11525.608342] Lustre: server umount lustre-OST0000 complete [11530.721529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11530.723693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11530.732086] Lustre: Skipped 1 previous similar message [11535.482214] Lustre: server umount lustre-MDT0000 complete [11538.872975] LustreError: 329382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121752 with bad export cookie 4468179505196218966 [11538.882730] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11538.885162] LustreError: 329382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11539.369560] Lustre: server umount lustre-MDT0001 complete [11547.554557] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 06:15:59 (1789121759) [11562.420229] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [11574.781590] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [11613.895869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11614.194071] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11614.272542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11614.361620] Lustre: lustre-MDT0000: new disk, initializing [11614.416572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11614.431548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11619.302542] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11629.851655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11629.972837] Lustre: 335727:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11630.030583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11630.044333] Lustre: Skipped 1 previous similar message [11630.206594] Lustre: lustre-MDT0001: new disk, initializing [11630.374814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11630.468315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11630.489890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11635.724402] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11644.160756] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11647.901574] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11655.126312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11655.359437] Lustre: lustre-OST0000: new disk, initializing [11655.366240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11655.386683] Lustre: 337469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11655.457680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11661.677915] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [11665.435987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11665.466651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11665.575562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11668.541362] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11696.618115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11696.638066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11696.651330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11696.654272] Lustre: Skipped 1 previous similar message [11701.217256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11701.227061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11701.242926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11702.453533] Lustre: server umount lustre-OST0000 complete [11706.851702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11706.860984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11706.891837] Lustre: Skipped 1 previous similar message [11706.920330] Lustre: Skipped 2 previous similar messages [11712.373939] Lustre: server umount lustre-MDT0000 complete [11716.551225] LustreError: 335718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121930 with bad export cookie 4468179505196219848 [11716.564537] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11716.577869] LustreError: 335718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11717.327066] Lustre: server umount lustre-MDT0001 complete [11729.283314] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [11731.911356] Key type lgssc unregistered [11732.436955] LNet: 339490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11732.448939] LNetError: 339490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11732.490228] LNet: Removed LNI 192.168.203.149@tcp [11733.456625] Key type .llcrypt unregistered [11733.460276] Key type ._llcrypt unregistered [11832.312301] Key type ._llcrypt registered [11832.314652] Key type .llcrypt registered [11832.390137] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [11832.600622] Key type .llcrypt unregistered [11832.604458] Key type ._llcrypt unregistered [11905.299628] Key type ._llcrypt registered [11905.301738] Key type .llcrypt registered [11905.414042] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [11905.711295] Key type .llcrypt unregistered [11905.713104] Key type ._llcrypt unregistered [11942.112962] Key type ._llcrypt registered [11942.116805] Key type .llcrypt registered [11942.214798] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [11942.607431] Key type .llcrypt unregistered [11942.614287] Key type ._llcrypt unregistered [11975.282740] Key type ._llcrypt registered [11975.285363] Key type .llcrypt registered [11975.385656] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [11975.700312] Key type .llcrypt unregistered [11975.703786] Key type ._llcrypt unregistered [12049.931974] Key type ._llcrypt registered [12049.934349] Key type .llcrypt registered [12050.026658] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [12050.275229] Key type .llcrypt unregistered [12050.277102] Key type ._llcrypt unregistered [12083.881916] Key type ._llcrypt registered [12083.887357] Key type .llcrypt registered [12084.037504] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [12084.386788] Key type .llcrypt unregistered [12084.389061] Key type ._llcrypt unregistered [12113.172986] Key type ._llcrypt registered [12113.178268] Key type .llcrypt registered [12113.333716] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [12113.726457] Key type .llcrypt unregistered [12113.728621] Key type ._llcrypt unregistered [12151.004895] Key type ._llcrypt registered [12151.007801] Key type .llcrypt registered [12151.163990] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [12151.462462] Key type .llcrypt unregistered [12151.465902] Key type ._llcrypt unregistered [12181.651394] Key type ._llcrypt registered [12181.661546] Key type .llcrypt registered [12181.780731] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [12182.036598] Key type .llcrypt unregistered [12182.040887] Key type ._llcrypt unregistered [12212.657309] Key type ._llcrypt registered [12212.662223] Key type .llcrypt registered [12212.776608] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [12212.972753] Key type .llcrypt unregistered [12212.975370] Key type ._llcrypt unregistered [12270.847246] Key type ._llcrypt registered [12270.850690] Key type .llcrypt registered [12271.027190] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [12271.320630] Key type .llcrypt unregistered [12271.324514] Key type ._llcrypt unregistered [12304.537676] Key type ._llcrypt registered [12304.540684] Key type .llcrypt registered [12304.703906] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [12305.096348] Key type .llcrypt unregistered [12305.107683] Key type ._llcrypt unregistered [12336.122869] Key type ._llcrypt registered [12336.126344] Key type .llcrypt registered [12336.266724] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [12336.505509] Key type .llcrypt unregistered [12336.507931] Key type ._llcrypt unregistered [12389.248210] Key type ._llcrypt registered [12389.254698] Key type .llcrypt registered [12389.347545] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [12404.529887] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [12405.524537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12405.553020] alg: No test for adler32 (adler32-zlib) [12406.695238] Lustre: Lustre: Build Version: 2.17.53_26_gdb05832 [12406.905981] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [12408.575167] Key type lgssc registered [12409.583638] Lustre: Echo OBD driver; http://www.lustre.org/ [12412.410515] alg: No test for zstdfast (zstdfast-lustre-generic) [12412.427254] alg: No test for zstdfast (zstdfast-lustre-scomp) [12445.652527] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12445.680175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12446.948327] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12447.026177] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12447.141959] Lustre: tst250fs-MDT0000: new disk, initializing [12447.202903] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12447.222415] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12451.576248] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [12462.059980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12462.205880] Lustre: 359771:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12462.287983] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12462.297923] Lustre: Skipped 1 previous similar message [12462.408120] Lustre: tst250fs-MDT0001: new disk, initializing [12462.517326] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12462.559312] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12462.576241] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12467.285806] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [12471.896573] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12478.223677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12478.529895] Lustre: tst250fs-OST0000: new disk, initializing [12478.534911] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12478.547219] Lustre: 361369:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12478.689573] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12478.913324] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12478.924331] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12479.091661] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12484.707221] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [12494.740252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12494.934191] Lustre: tst250fs-OST0001: new disk, initializing [12494.942831] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12494.951460] Lustre: 362225:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12495.075990] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12500.575201] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [12501.080441] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12501.099293] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12501.228405] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12505.846307] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12508.279372] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12508.284988] Lustre: Skipped 2 previous similar messages [12527.073540] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12527.089785] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12527.092652] Lustre: Skipped 1 previous similar message [12527.102642] Lustre: Skipped 3 previous similar messages [12532.200757] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12532.213461] Lustre: Skipped 3 previous similar messages [12537.323231] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12537.331809] Lustre: Skipped 3 previous similar messages [12540.384579] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12540.706396] Lustre: server umount tst250fs-MDT0000 complete [12544.064683] LustreError: 362224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122757 with bad export cookie 14920376977102613571 [12544.068067] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12544.081178] LustreError: 362224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12544.592816] Lustre: server umount tst250fs-MDT0001 complete [12558.782812] Lustre: server umount tst250fs-OST0000 complete [12563.937513] Lustre: 357948:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789122761/real 1789122761] req@ffff8e5610959f80 x1876031040459008/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789122777 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12563.987889] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12564.035034] Lustre: Skipped 2 previous similar messages [12565.516456] Lustre: server umount tst250fs-OST0001 complete [12586.191889] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [12596.358899] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [12630.787523] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 06:34:02 (1789122842) [12632.260178] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12648.170587] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [12657.277180] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [12686.651735] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12337 sec ========= 06:34:58 (1789122898) [12688.408893] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:34:59 (1789122899) === [12691.978398] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:35:03 (1789122903) === [12736.427697] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [12739.582604] Key type lgssc unregistered [12739.928892] LNet: 371295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12739.936540] LNetError: 371295:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12739.948257] LNet: Removed LNI 192.168.203.149@tcp [12741.126277] Key type .llcrypt unregistered [12741.128695] Key type ._llcrypt unregistered