[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1487448320 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002426] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008669] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011152] LSM: Security Framework initializing [ 0.012066] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014076] *** VALIDATE selinux *** [ 0.023265] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028527] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032111] *** VALIDATE tmpfs *** [ 0.034087] *** VALIDATE proc *** [ 0.035188] *** VALIDATE cgroup *** [ 0.036009] *** VALIDATE cgroup2 *** [ 0.037296] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039201] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040081] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041035] Spectre V2 : User space: Vulnerable [ 0.042023] Speculative Store Bypass: Vulnerable [ 0.046573] debug: unmapping init [mem 0xffffffffb5459000-0xffffffffb5460fff] [ 0.048159] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049935] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050023] ... version: 2 [ 0.051024] ... bit width: 48 [ 0.052031] ... generic registers: 4 [ 0.053019] ... value mask: 0000ffffffffffff [ 0.054022] ... max period: 00007fffffffffff [ 0.055023] ... fixed-purpose events: 3 [ 0.056021] ... event mask: 000000070000000f [ 0.057415] rcu: Hierarchical SRCU implementation. [ 0.060352] smp: Bringing up secondary CPUs ... [ 0.061670] x86: Booting SMP configuration: [ 0.062063] .... node #0, CPUs: #1 #2 #3 [ 0.071464] smp: Brought up 1 node, 4 CPUs [ 0.073015] smpboot: Max logical packages: 1 [ 0.074012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.109764] node 0 deferred pages initialised in 33ms [ 0.113166] devtmpfs: initialized [ 0.114278] x86/mm: Memory block size: 128MB [ 0.117401] gcov: version magic: 0x41383552 [ 0.119383] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.120547] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.121279] pinctrl core: initialized pinctrl subsystem [ 0.122326] [ 0.122872] ************************************************************* [ 0.123013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124012] ** ** [ 0.125012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.126018] ** ** [ 0.127011] ** This means that this kernel is built to expose internal ** [ 0.128015] ** IOMMU data structures, which may compromise security on ** [ 0.129023] ** your system. ** [ 0.130012] ** ** [ 0.131012] ** If you see this message and you are not debugging the ** [ 0.132013] ** kernel, report this immediately to your vendor! ** [ 0.133015] ** ** [ 0.134013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135012] ************************************************************* [ 0.137265] NET: Registered protocol family 16 [ 0.138455] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.139066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.140064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.141873] cpuidle: using governor menu [ 0.145425] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.146437] PCI: Using configuration type 1 for base access [ 0.147133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.158087] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.164020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.171353] cryptd: max_cpu_qlen set to 1000 [ 0.184368] ACPI: Added _OSI(Module Device) [ 0.187016] ACPI: Added _OSI(Processor Device) [ 0.188000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.188000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.188000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207431] ACPI: Interpreter enabled [ 0.211077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214057] ACPI: Using IOAPIC for interrupt routing [ 0.215132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234572] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.236043] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.241456] acpiphp: Slot [2] registered [ 0.242199] acpiphp: Slot [5] registered [ 0.243236] acpiphp: Slot [6] registered [ 0.244257] acpiphp: Slot [7] registered [ 0.245180] acpiphp: Slot [8] registered [ 0.246251] acpiphp: Slot [9] registered [ 0.247166] acpiphp: Slot [10] registered [ 0.248135] acpiphp: Slot [3] registered [ 0.249143] acpiphp: Slot [4] registered [ 0.250123] acpiphp: Slot [11] registered [ 0.251279] acpiphp: Slot [12] registered [ 0.252137] acpiphp: Slot [13] registered [ 0.253289] acpiphp: Slot [14] registered [ 0.254140] acpiphp: Slot [15] registered [ 0.256531] acpiphp: Slot [16] registered [ 0.257140] acpiphp: Slot [17] registered [ 0.258091] acpiphp: Slot [18] registered [ 0.259086] acpiphp: Slot [19] registered [ 0.260153] acpiphp: Slot [20] registered [ 0.261239] acpiphp: Slot [21] registered [ 0.262145] acpiphp: Slot [22] registered [ 0.263256] acpiphp: Slot [23] registered [ 0.264362] acpiphp: Slot [24] registered [ 0.265129] acpiphp: Slot [25] registered [ 0.266098] acpiphp: Slot [26] registered [ 0.267099] acpiphp: Slot [27] registered [ 0.268147] acpiphp: Slot [28] registered [ 0.269131] acpiphp: Slot [29] registered [ 0.270563] acpiphp: Slot [30] registered [ 0.271123] acpiphp: Slot [31] registered [ 0.272068] PCI host bridge to bus 0000:00 [ 0.273029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.274059] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.275872] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.276028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.277023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.278028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279256] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.280984] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.282261] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.290017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.293076] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.294025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.295029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.296031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.297577] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.300488] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.301035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.302808] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.304014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.311021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.313013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.316807] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.319014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.322016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.329026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.335342] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.338018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.341019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.348018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.353963] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.356022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.359013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.366019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.372361] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.375017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.378018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.385017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.391100] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.394015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.397015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.404017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.411105] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.414014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.417014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.424020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.431062] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.432436] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.433357] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.435404] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.439237] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.448026] iommu: Default domain type: Passthrough [ 0.453681] SCSI subsystem initialized [ 0.454130] ACPI: bus type USB registered [ 0.455314] usbcore: registered new interface driver usbfs [ 0.456153] usbcore: registered new interface driver hub [ 0.457077] usbcore: registered new device driver usb [ 0.458191] pps_core: LinuxPPS API ver. 1 registered [ 0.459012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.460056] PTP clock support registered [ 0.466074] EDAC MC: Ver: 3.0.0 [ 0.493144] PCI: Using ACPI for IRQ routing [ 0.495164] NetLabel: Initializing [ 0.496011] NetLabel: domain hash size = 128 [ 0.497009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.498150] NetLabel: unlabeled traffic allowed by default [ 0.499861] vgaarb: loaded [ 0.501469] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.502011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.505633] clocksource: Switched to clocksource kvm-clock [ 0.946274] VFS: Disk quotas dquot_6.6.0 [ 0.948301] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.950390] *** VALIDATE ramfs *** [ 0.951629] *** VALIDATE hugetlbfs *** [ 0.953056] pnp: PnP ACPI init [ 0.955749] pnp: PnP ACPI: found 6 devices [ 0.970360] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.972703] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.977258] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.978658] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.983617] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.986560] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.990740] NET: Registered protocol family 2 [ 0.996363] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.003625] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.012326] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.024699] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.027647] TCP: Hash tables configured (established 65536 bind 65536) [ 1.032544] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.039453] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.043503] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.050721] NET: Registered protocol family 1 [ 1.052638] RPC: Registered named UNIX socket transport module. [ 1.057835] RPC: Registered udp transport module. [ 1.060806] RPC: Registered tcp transport module. [ 1.066183] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.070855] NET: Registered protocol family 44 [ 1.074938] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.078903] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.085196] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.086938] PCI: CLS 0 bytes, default 64 [ 1.092879] Unpacking initramfs... [ 3.534309] hrtimer: interrupt took 27660219 ns [ 5.550869] debug: unmapping init [mem 0xffff9174fcc54000-0xffff9174fffbffff] [ 5.557226] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.558994] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.562513] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.797560] Initialise system trusted keyrings [ 6.804488] Key type blacklist registered [ 6.811829] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.820853] zbud: loaded [ 6.823894] *** VALIDATE nfs *** [ 6.825391] *** VALIDATE nfs4 *** [ 6.826941] pstore: using deflate compression [ 6.833505] Platform Keyring initialized [ 7.075441] NET: Registered protocol family 38 [ 7.111444] Key type asymmetric registered [ 7.116339] Asymmetric key parser 'x509' registered [ 7.125145] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.147479] io scheduler mq-deadline registered [ 7.148910] io scheduler kyber registered [ 7.167310] io scheduler bfq registered [ 7.176659] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.189832] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.200918] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.212853] ACPI: Power Button [PWRF] [ 7.234893] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.262173] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.299199] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.322668] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.368583] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.427973] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.530171] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.549436] Non-volatile memory driver v1.3 [ 7.554550] Linux agpgart interface v0.103 [ 7.730064] virtio_blk virtio1: [vda] 134152 512-byte logical blocks (68.7 MB/65.5 MiB) [ 7.748566] vda: detected capacity change from 0 to 68685824 [ 7.887345] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.889944] vdb: detected capacity change from 0 to 1073741824 [ 7.913863] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.916647] vdc: detected capacity change from 0 to 2621440000 [ 7.944434] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.967507] vdd: detected capacity change from 0 to 2621440000 [ 8.032499] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.035691] vde: detected capacity change from 0 to 4294967296 [ 8.050675] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.053584] vdf: detected capacity change from 0 to 4294967296 [ 8.059574] libphy: Fixed MDIO Bus: probed [ 8.066245] usbcore: registered new interface driver usbserial_generic [ 8.069284] usbserial: USB Serial support registered for generic [ 8.071804] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.075858] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.081395] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.083747] mousedev: PS/2 mouse device common for all mice [ 8.087525] rtc_cmos 00:05: RTC can wake from S4 [ 8.093969] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.097440] rtc_cmos 00:05: registered as rtc0 [ 8.097558] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.097608] intel_pstate: CPU model not supported [ 8.101826] hid: raw HID events driver (C) Jiri Kosina [ 8.112200] usbcore: registered new interface driver usbhid [ 8.114804] usbhid: USB HID core driver [ 8.116983] drop_monitor: Initializing network drop monitor service [ 8.119450] Initializing XFRM netlink socket [ 8.122727] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.194625] NET: Registered protocol family 10 [ 8.309777] Segment Routing with IPv6 [ 8.313170] NET: Registered protocol family 17 [ 8.316272] mpls_gso: MPLS GSO support [ 8.318180] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.330558] RAS: Correctable Errors collector initialized. [ 8.333353] AVX version of gcm_enc/dec engaged. [ 8.335216] AES CTR mode by8 optimization enabled [ 8.712328] sched_clock: Marking stable (8712094163, 0)->(11773109612, -3061015449) [ 8.720630] registered taskstats version 1 [ 8.729831] Loading compiled-in X.509 certificates [ 8.732079] zswap: loaded using pool lzo/zbud [ 8.759692] Key type big_key registered [ 8.774726] Key type encrypted registered [ 8.777300] ima: No TPM chip found, activating TPM-bypass! [ 8.779410] ima: Allocated hash algorithm: sha1 [ 8.786591] ima: No architecture policies found [ 8.788644] evm: Initialising EVM extended attributes: [ 8.790458] evm: security.selinux [ 8.791459] evm: security.ima [ 8.792403] evm: security.capability [ 8.793546] evm: HMAC attrs: 0x1 [ 8.797544] rtc_cmos 00:05: setting system clock to 2026-05-08 22:07:43 UTC (1778278063) [ 8.805241] debug: unmapping init [mem 0xffffffffb6403000-0xffffffffb65fffff] [ 8.808016] debug: unmapping init [mem 0xffffffffb5182000-0xffffffffb5458fff] [ 8.816184] Write protecting the kernel read-only data: 28672k [ 8.822396] debug: unmapping init [mem 0xffffffffb3803000-0xffffffffb39fffff] [ 8.826174] debug: unmapping init [mem 0xffffffffb4114000-0xffffffffb41fffff] [ 9.020995] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.034887] systemd[1]: Detected virtualization kvm. [ 9.037395] systemd[1]: Detected architecture x86-64. [ 9.039848] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.070823] systemd[1]: No hostname configured. [ 9.072875] systemd[1]: Set hostname to . [ 9.075617] random: systemd: uninitialized urandom read (16 bytes read) [ 9.079754] systemd[1]: Initializing machine ID from random generator. [ 9.433601] random: ln: uninitialized urandom read (6 bytes read) [ 10.434579] random: systemd: uninitialized urandom read (16 bytes read) [ 10.437427] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 10.448201] random: systemd: uninitialized urandom read (16 bytes read) [ 10.451386] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 10.468723] random: systemd: uninitialized urandom read (16 bytes read) [ 10.476886] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 15.468654] device-mapper: uevent: version 1.0.3 [ 15.479187] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Mounting Kernel Configuration File System... Starting dracut initqueue hook... [ OK ] Mounted Kernel Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [ 21.309431] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. [ 21.529089] virtio_net virtio0 ens2: renamed from eth0 [ 22.113102] scsi host0: ata_piix [ 23.740097] scsi host1: ata_piix [ 23.775417] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 23.820740] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 30.516647] random: crng init done [ 30.518115] random: 5 urandom warning(s) missed due to ratelimiting [ 35.857871] dracut-initqueue[577]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 38.734250] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 44.796955] printk: systemd: 26 output lines suppressed due to ratelimiting [ 46.565484] SELinux: Disabled at runtime. [ 46.848281] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 46.892656] systemd[1]: Detected virtualization kvm. [ 46.909422] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 49.647180] systemd[1]: initrd-switch-root.service: Succeeded. [ 49.654974] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 49.685806] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 49.700977] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 49.709287] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 49.751880] systemd[1]: Starting Journal Service... Starting Journal Service... [ 49.791619] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ 50.665862] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 52.464650] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 54.132402] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 54.310921] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 55.470578] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 55.727687] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit) [** ] A start job is running for Configur…only root support (15s / no limit) [* ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit) [*** ] A start job is running for Configur…only root support (16s / no limit)[ 66.434361] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (17s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit) [ ***] A start job is running for Configur…only root support (18s / no limit)[ 68.079089] NFS: Registering the id_resolver key type [ 68.087475] Key type id_resolver registered [ 68.102816] Key type id_legacy registered [ **] A start job is running for Configur…only root support (18s / no limit) [ *] A start job is running for Configur…only root support (19s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Login Service. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg323-server login: [ 130.884396] libcfs: loading out-of-tree module taints kernel. [ 130.936759] Key type ._llcrypt registered [ 130.938270] Key type .llcrypt registered [ 131.063257] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 146.611270] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 148.132561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 148.159166] alg: No test for adler32 (adler32-zlib) [ 149.431765] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 150.357058] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 152.127159] Key type lgssc registered [ 153.752689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 168.583545] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 204.593597] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 216.145480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 216.166205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.425624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 217.467501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 217.556948] Lustre: lustre-MDT0000: new disk, initializing [ 217.631249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.660455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 221.479835] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.392432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 232.456204] Lustre: 6491:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 232.502476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 232.506249] Lustre: Skipped 1 previous similar message [ 232.591908] Lustre: lustre-MDT0001: new disk, initializing [ 232.652051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.689687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 232.697312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 235.867795] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.951697] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 247.868311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.113831] Lustre: lustre-OST0000: new disk, initializing [ 248.118709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.183213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.693428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 248.702896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 248.783138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 253.055906] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.387866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 263.475314] Lustre: lustre-OST0001: new disk, initializing [ 263.478255] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 263.559169] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 268.286846] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.863588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 270.873893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 270.907161] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 278.453812] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 282.468860] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 292.921583] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing check_logdir /tmp/testlogs/ [ 296.470389] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing yml_node [ 299.459811] Lustre: DEBUG MARKER: Client: 2.17.52.6 [ 301.377172] Lustre: DEBUG MARKER: MDS: 2.17.52.6 [ 303.825694] Lustre: DEBUG MARKER: OSS: 2.17.52.6 [ 305.327033] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri May 8 18:12:38 EDT 2026 [ 319.522444] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 320.591535] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 335.328386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 335.338485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.351522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.377283] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.378858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.398565] Lustre: Skipped 2 previous similar messages [ 337.409396] Lustre: Skipped 3 previous similar messages [ 340.649945] Lustre: server umount lustre-MDT0000 complete [ 343.136851] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778278397 with bad export cookie 1592462302098886844 [ 343.138268] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.148637] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 347.621631] LustreError: 6498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.623043] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.623690] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 347.641636] LustreError: 6498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 347.664291] Lustre: Skipped 1 previous similar message [ 348.640222] LustreError: 6500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.651231] LustreError: 6500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 349.395131] Lustre: server umount lustre-MDT0001 complete [ 352.455183] Lustre: server umount lustre-OST0000 complete [ 355.544650] Lustre: server umount lustre-OST0001 complete [ 360.775495] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 368.255817] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 397.645632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.867579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 397.890556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 397.968966] Lustre: lustre-MDT0000: new disk, initializing [ 398.088535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.110698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.785162] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 410.201121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.319265] Lustre: 14963:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 410.421597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 410.425181] Lustre: Skipped 1 previous similar message [ 410.491185] Lustre: lustre-MDT0001: new disk, initializing [ 410.587490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 410.639540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 410.655605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 414.334217] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 420.286432] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 423.626745] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 429.270203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.488753] Lustre: lustre-OST0000: new disk, initializing [ 429.492817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.571127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.460528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 431.478033] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.590993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 433.700944] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 440.283726] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 443.711683] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 443.893563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 447.112774] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 447.257580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 452.066068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 452.068364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.071092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.071103] Lustre: Skipped 3 previous similar messages [ 452.072583] LustreError: Skipped 1 previous similar message [ 452.104648] Lustre: Skipped 1 previous similar message [ 454.773611] Lustre: server umount lustre-OST0000 complete [ 457.185854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.188371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 457.199907] Lustre: Skipped 1 previous similar message [ 457.207402] Lustre: Skipped 2 previous similar messages [ 463.164211] Lustre: server umount lustre-MDT0000 complete [ 465.288530] LustreError: 14957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778278519 with bad export cookie 1592462302098888377 [ 465.293835] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 465.294943] LustreError: 14957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 465.475801] Lustre: server umount lustre-MDT0001 complete [ 468.583784] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:15:22 (1778278522) [ 479.078290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.381570] LustreError: 19431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.433602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.714698] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 484.834373] LustreError: 19432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.778351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.654890] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 493.150790] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 495.342173] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 496.717784] Lustre: Failing over lustre-MDT0000 [ 496.855856] Lustre: server umount lustre-MDT0000 complete [ 497.121382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 497.132129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.142655] LustreError: 19432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.384040] LustreError: 19428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.391294] LustreError: 19428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 511.834207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.899553] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.057127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 513.060761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 513.071199] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 513.758587] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 517.638150] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 519.767480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.772903] Lustre: Skipped 1 previous similar message [ 519.775234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 519.777972] Lustre: Skipped 3 previous similar messages [ 525.958394] Lustre: server umount lustre-MDT0000 complete [ 527.636113] LustreError: 19412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778278582 with bad export cookie 1592462302098889791 [ 527.638644] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.642734] LustreError: 19412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 527.783591] Lustre: server umount lustre-MDT0001 complete [ 532.151333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.471869] LustreError: 22423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.859821] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 539.888510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.499265] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 545.720179] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 547.956201] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 549.226135] Lustre: Failing over lustre-MDT0000 [ 549.351980] Lustre: server umount lustre-MDT0000 complete [ 564.363522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.438759] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 564.583791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.589584] Lustre: Skipped 4 previous similar messages [ 565.602618] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 565.613038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 565.618887] Lustre: Skipped 1 previous similar message [ 565.629493] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 566.590469] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 570.875310] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 575.969197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 575.969634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.983740] Lustre: Skipped 3 previous similar messages [ 575.989918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 575.993715] Lustre: Skipped 3 previous similar messages [ 579.489469] Lustre: server umount lustre-MDT0000 complete [ 580.978215] LustreError: 22405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778278635 with bad export cookie 1592462302098890729 [ 580.981891] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 580.983402] LustreError: 22405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 581.109797] Lustre: server umount lustre-MDT0001 complete [ 584.346618] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:17:18 (1778278638) [ 590.056855] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 593.491279] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 608.790602] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 613.247692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.360396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 613.377762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 613.422623] Lustre: lustre-MDT0000: new disk, initializing [ 613.472593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 615.086781] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 619.908723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.957536] Lustre: 28394:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 620.074447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 621.579794] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 625.072124] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 626.584651] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 629.368814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 630.680514] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 630.711265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 630.720279] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 631.941259] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 635.196059] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 637.794792] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 645.278889] Lustre: server umount lustre-OST0000 complete [ 650.719764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 650.722986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.728109] Lustre: Skipped 1 previous similar message [ 650.729963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 656.378665] Lustre: server umount lustre-MDT0000 complete [ 657.684345] LustreError: 28385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778278712 with bad export cookie 1592462302098891135 [ 657.688269] LustreError: 28385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 657.688752] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 668.184911] Lustre: server umount lustre-MDT0001 complete [ 671.987094] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 673.085560] Key type lgssc unregistered [ 673.222608] LNet: 31961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.225859] LNetError: 31961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 673.234466] LNet: Removed LNI 192.168.203.123@tcp [ 673.583140] Key type .llcrypt unregistered [ 673.584551] Key type ._llcrypt unregistered [ 683.376159] Key type ._llcrypt registered [ 683.377820] Key type .llcrypt registered [ 683.422642] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 683.770834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 683.824881] alg: No test for adler32 (adler32-zlib) [ 684.695154] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 684.780057] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 686.359175] Key type lgssc registered [ 686.772785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 689.384800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 689.391508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.554223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.852799] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 694.623933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.743564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 696.068459] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 697.798226] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 699.080584] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 701.335046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 701.455631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 703.321017] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 704.550103] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 705.137889] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 721.194938] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:19:35 (1778278775) [ 721.828426] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 722.405498] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 724.496982] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:19:38 (1778278778) [ 727.195819] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:19:41 (1778278781) [ 727.532588] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 730.063753] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 730.066729] Lustre: Skipped 2 previous similar messages [ 730.597578] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:19:44 (1778278784) [ 737.322315] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 737.324824] Lustre: Skipped 10 previous similar messages [ 738.173385] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:19:52 (1778278792) [ 739.050786] Lustre: *** cfs_fail_loc=131b, val=2*** [ 740.179920] Lustre: *** cfs_fail_loc=131b, val=2*** [ 742.934642] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:19:57 (1778278797) [ 747.433613] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 747.436449] Lustre: Skipped 7 previous similar messages [ 747.924609] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:20:02 (1778278802) [ 755.679991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 755.684432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.690976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 756.704474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 756.707589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 756.707717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 757.712864] Lustre: server umount lustre-OST0000 complete [ 761.311984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 761.314940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.320440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 761.322468] Lustre: Skipped 1 previous similar message [ 765.018136] Lustre: server umount lustre-MDT0000 complete [ 766.191092] LustreError: 38030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778278820 with bad export cookie 18017510421453860446 [ 766.194229] LustreError: 38030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 766.195138] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 766.309030] Lustre: server umount lustre-MDT0001 complete [ 769.594306] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 770.541238] Key type lgssc unregistered [ 770.661264] LNet: 40830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.663452] LNetError: 40830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.674243] LNet: Removed LNI 192.168.203.123@tcp [ 770.929921] Key type .llcrypt unregistered [ 770.931059] Key type ._llcrypt unregistered [ 776.193230] Key type ._llcrypt registered [ 776.194246] Key type .llcrypt registered [ 776.224264] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 781.924110] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 782.264309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 782.318234] alg: No test for adler32 (adler32-zlib) [ 783.171746] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 783.246271] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 784.823119] Key type lgssc registered [ 785.135724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 795.427493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 795.432734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 796.519365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 796.531757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 796.566736] Lustre: lustre-MDT0000: new disk, initializing [ 796.589981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 796.596603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 797.884395] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 801.966643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.007767] Lustre: 43920:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 802.031446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 802.033301] Lustre: Skipped 1 previous similar message [ 802.072568] Lustre: lustre-MDT0001: new disk, initializing [ 802.106348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 802.120864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 802.124775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 803.400331] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 806.456283] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 807.733567] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 810.024266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 810.128852] Lustre: lustre-OST0000: new disk, initializing [ 810.130940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 810.163102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.495147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 810.500291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 810.529474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 812.129247] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 815.253623] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 816.607711] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 816.677169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 818.073785] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 818.152925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 820.706539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 820.707951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.709216] LustreError: Skipped 1 previous similar message [ 820.709867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 820.713023] Lustre: Skipped 1 previous similar message [ 824.911460] Lustre: server umount lustre-OST0000 complete [ 827.871744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 827.873806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.878708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 827.880608] Lustre: Skipped 1 previous similar message [ 830.944650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 830.945324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 832.283631] Lustre: server umount lustre-MDT0000 complete [ 833.483102] LustreError: 45661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778278888 with bad export cookie 1228181547514122964 [ 833.486610] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 833.486935] LustreError: 45661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 833.595425] Lustre: server umount lustre-MDT0001 complete [ 836.660610] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:21:30 (1778278890) [ 838.880794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.043451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 840.303224] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 843.091476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 843.212363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 844.625021] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 846.392597] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 847.605870] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 849.933969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 852.017547] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 853.772906] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 866.059196] LustreError: 50186:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 866.362594] Lustre: Modifying parameter general.timeout=1 in log params [ 867.590450] Lustre: Modifying parameter general.timeout=5 in log params [ 867.592284] Lustre: Skipped 3 previous similar messages [ 869.772616] Lustre: Modifying parameter general.timeout=12 in log params [ 869.774805] Lustre: Skipped 6 previous similar messages [ 873.789937] Lustre: Modifying parameter general.timeout=25 in log params [ 873.792886] Lustre: Skipped 12 previous similar messages [ 881.872531] Lustre: Modifying parameter general.timeout=51 in log params [ 881.875359] Lustre: Skipped 25 previous similar messages [ 897.889799] Lustre: Modifying parameter general.timeout=105 in log params [ 897.891532] Lustre: Skipped 53 previous similar messages [ 908.698324] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:22:42 (1778278962) [ 909.625818] Lustre: 57082:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule8 rate=1.5GiB/s minrate=512MiB/s (mode = 2) failed: rc = -17 [ 910.836721] Lustre: 57280:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule6 nid={10.10.2.20@tcp} minrate=100M rate=1G (mode = 2) failed: rc = -17 [ 910.841128] Lustre: 57280:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 913.259110] Lustre: 57687:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 913.263809] Lustre: 57687:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 3 previous similar messages [ 930.393942] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} minrate=50 rate=500 in log params [ 930.397460] Lustre: Skipped 65 previous similar messages [ 932.662304] Lustre: 60736:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 932.665723] Lustre: 60736:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 6 previous similar messages [ 932.668045] LustreError: 60736:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 934.010373] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:23:08 (1778278988) [ 938.465414] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.466610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 938.470181] Lustre: Skipped 2 previous similar messages [ 938.473727] Lustre: Skipped 3 previous similar messages [ 943.585860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.589054] Lustre: Skipped 2 previous similar messages [ 943.705029] Lustre: server umount lustre-MDT0000 complete [ 944.946937] LustreError: 47954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778278999 with bad export cookie 1228181547514123832 [ 944.949762] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 944.950414] LustreError: 47954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 945.063925] Lustre: server umount lustre-MDT0001 complete [ 956.629716] Lustre: server umount lustre-OST0000 complete [ 965.426740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.530360] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 965.647335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.650102] Lustre: Skipped 1 previous similar message [ 967.139644] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 970.232844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.270715] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 970.281797] Lustre: 63263:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 971.897446] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 973.820477] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 975.115881] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 977.448481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 977.524211] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 979.673340] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 981.554603] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 985.057168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 985.058477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.059825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 985.060289] LustreError: Skipped 1 previous similar message [ 985.064686] Lustre: Skipped 1 previous similar message [ 990.163705] Lustre: server umount lustre-OST0000 complete [ 995.296719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 995.301509] Lustre: Skipped 1 previous similar message [ 997.594507] Lustre: server umount lustre-MDT0000 complete [ 998.853158] LustreError: 63255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778279053 with bad export cookie 1228181547514126429 [ 998.856587] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 998.858971] LustreError: 63255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 998.969330] Lustre: server umount lustre-MDT0001 complete [ 1002.655628] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1003.704954] Key type lgssc unregistered [ 1003.838341] LNet: 66676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.841246] LNetError: 66676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.847346] LNet: Removed LNI 192.168.203.123@tcp [ 1004.144811] Key type .llcrypt unregistered [ 1004.146083] Key type ._llcrypt unregistered [ 1013.270793] Key type ._llcrypt registered [ 1013.272033] Key type .llcrypt registered [ 1013.309516] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1013.655451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1013.688650] alg: No test for adler32 (adler32-zlib) [ 1014.542614] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 1014.631644] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1016.207105] Key type lgssc registered [ 1016.554102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1019.189433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1019.196454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.390283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.721684] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1024.566592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.766495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1026.185348] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1028.005343] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1029.178661] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1031.466228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1031.679720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.618052] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1035.591743] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1053.933660] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1060.831855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1060.834109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.838029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1061.344216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1061.345926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.347268] Lustre: Skipped 1 previous similar message [ 1061.350861] Lustre: Skipped 1 previous similar message [ 1062.551977] Lustre: server umount lustre-MDT0000 complete [ 1063.776983] LustreError: 68029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778279118 with bad export cookie 17133827102640592591 [ 1063.780596] LustreError: 68029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1063.780672] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1063.894462] Lustre: server umount lustre-MDT0001 complete [ 1074.393501] Lustre: server umount lustre-OST0000 complete [ 1083.177334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.277221] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1083.287164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1083.376312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.696914] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1087.512640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1087.547215] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1087.557869] Lustre: 72602:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1087.738551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1089.067612] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1090.898402] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1092.059800] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1094.356939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1094.455945] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1096.584647] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1098.430000] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1098.970088] Lustre: 74838:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1099.585537] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1103.330432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.331214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.335692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1103.337640] LustreError: Skipped 1 previous similar message [ 1103.340652] Lustre: Skipped 1 previous similar message [ 1107.016955] Lustre: server umount lustre-OST0000 complete [ 1108.448675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.448697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1108.449278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1108.454480] Lustre: Skipped 2 previous similar messages [ 1113.568089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1113.569762] Lustre: Skipped 2 previous similar messages [ 1114.455116] Lustre: server umount lustre-MDT0000 complete [ 1115.769198] LustreError: 74840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778279170 with bad export cookie 17133827102640593403 [ 1115.772894] LustreError: 74840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1115.773098] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1115.900497] Lustre: server umount lustre-MDT0001 complete [ 1119.603914] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1120.668794] Key type lgssc unregistered [ 1120.799394] LNet: 76028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.803759] LNetError: 76028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1120.814331] LNet: Removed LNI 192.168.203.123@tcp [ 1121.129185] Key type .llcrypt unregistered [ 1121.130870] Key type ._llcrypt unregistered [ 1129.930917] Key type ._llcrypt registered [ 1129.932269] Key type .llcrypt registered [ 1129.966644] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1130.315102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1130.339337] alg: No test for adler32 (adler32-zlib) [ 1131.196158] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 1131.278614] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1132.863184] Key type lgssc registered [ 1133.232623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1136.000597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1136.007550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.198966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.590780] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1141.577041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.799835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1143.163750] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1144.982600] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1146.207431] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1148.411384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1148.613174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1150.423490] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1152.165882] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1169.945615] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1170.507860] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1170.509583] Lustre: Skipped 1 previous similar message [ 1171.638124] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1171.639873] Lustre: Skipped 3 previous similar messages [ 1173.889062] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1173.891227] Lustre: Skipped 7 previous similar messages [ 1177.936511] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1177.938792] Lustre: Skipped 13 previous similar messages [ 1186.051138] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1186.052926] Lustre: Skipped 28 previous similar messages [ 1200.336881] Lustre: 85175:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1201.166560] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:27:35 (1778279255) [ 1201.676963] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1202.259111] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:27:36 (1778279256) [ 1204.194677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1204.194687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.197697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1204.201851] Lustre: Skipped 1 previous similar message [ 1204.206492] Lustre: Skipped 1 previous similar message [ 1209.292304] Lustre: server umount lustre-OST0000 complete [ 1213.407841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1213.410250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.414057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1214.432252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1214.432947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1216.735268] Lustre: server umount lustre-MDT0000 complete [ 1217.959732] LustreError: 78122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778279272 with bad export cookie 1586480300362163128 [ 1217.964076] LustreError: 78122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1217.964342] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1218.088824] Lustre: server umount lustre-MDT0001 complete [ 1221.449979] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1222.374247] Key type lgssc unregistered [ 1222.491392] LNet: 86403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.495117] LNetError: 86403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.505308] LNet: Removed LNI 192.168.203.123@tcp [ 1222.789374] Key type .llcrypt unregistered [ 1222.790525] Key type ._llcrypt unregistered [ 1224.323817] Key type ._llcrypt registered [ 1224.325033] Key type .llcrypt registered [ 1224.365900] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1225.674387] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules [ 1226.059189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1226.071785] alg: No test for adler32 (adler32-zlib) [ 1226.922691] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 1227.004075] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1228.583168] Key type lgssc registered [ 1228.981231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1229.352338] LustreError: 87523:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 1236.989711] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1240.044336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1250.567700] LustreError: 87523:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1250.577500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.772308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.072950] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1255.797041] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:28:30 (1778279310) [ 1257.259681] LustreError: 88560:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1257.262867] LustreError: 88560:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1257.265621] LustreError: 88560:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1257.376200] Lustre: server umount lustre-MDT0000 complete [ 1261.646325] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 1262.590665] Key type lgssc unregistered [ 1262.712391] LNet: 90019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.715556] LNetError: 90019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.725281] LNet: Removed LNI 192.168.203.123@tcp [ 1263.003296] Key type .llcrypt unregistered [ 1263.004509] Key type ._llcrypt unregistered [ 1269.973783] Key type ._llcrypt registered [ 1269.975305] Key type .llcrypt registered [ 1270.019116] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1270.501171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1270.527720] alg: No test for adler32 (adler32-zlib) [ 1271.384256] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 1271.472227] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 1273.055163] Key type lgssc registered [ 1273.431438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1276.175332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1276.181730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.367981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1278.735681] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1281.549924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.792340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1283.227992] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1285.077278] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1286.380251] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1288.644165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1288.880458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1290.792490] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1292.487915] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1329.075820] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:29:43 (1778279383) [ 1329.650362] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1330.262326] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:29:44 (1778279384) [ 1333.216032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1333.218347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.222163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.240714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.243746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.249232] Lustre: Skipped 2 previous similar messages [ 1337.426916] Lustre: server umount lustre-MDT0000 complete [ 1338.628178] LustreError: 91175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778279393 with bad export cookie 6409679532577784713 [ 1338.632156] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1338.632161] LustreError: 91175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1338.757621] Lustre: server umount lustre-MDT0001 complete [ 1350.283228] Lustre: server umount lustre-OST0000 complete [ 1354.043695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1354.194883] LustreError: 95151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.201938] LustreError: 95151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1354.250266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.558727] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1358.488345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.736587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1360.062513] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1361.895124] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1363.137674] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1367.619220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1367.725689] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1367.728648] LustreError: 95141:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1367.743612] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1367.746677] LustreError: 97005:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 1367.748938] LustreError: 97005:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1367.751079] LustreError: 97005:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1367.776410] Lustre: server umount lustre-OST0000 complete [ 1367.777680] LustreError: 97005:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1370.600874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.672461] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1370.675259] LustreError: 95141:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1370.679610] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1370.682628] LustreError: 97469:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 1370.685453] LustreError: 97469:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1370.687476] LustreError: 97469:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1370.708492] Lustre: server umount lustre-OST0000 complete [ 1370.709891] LustreError: 97469:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1373.907365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1373.977791] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1373.980464] Lustre: Found index 0 for lustre-OST0000, updating log [ 1373.982622] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1374.015813] Lustre: lustre-OST0000: new disk, initializing [ 1374.017707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1374.108970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.108711] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1377.360809] Lustre: Failing over lustre-OST0000 [ 1377.409461] Lustre: server umount lustre-OST0000 complete [ 1379.295728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1379.298489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.302991] Lustre: Skipped 1 previous similar message [ 1379.304973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1383.392931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1383.393883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.800358] Lustre: server umount lustre-MDT0000 complete [ 1386.101225] LustreError: 95132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778279440 with bad export cookie 6409679532577787443 [ 1386.104086] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1386.106509] LustreError: 95132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1386.222606] Lustre: server umount lustre-MDT0001 complete [ 1388.997836] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:30:43 (1778279443) [ 1394.241960] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1398.006590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.153517] LustreError: 100308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.181573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.463405] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1402.185557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.702460] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1404.618953] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1406.902877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1409.015232] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1411.801656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1411.831466] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1411.861202] Lustre: lustre-OST0001: new disk, initializing [ 1411.862938] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1413.752649] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1416.701464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1416.706022] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1416.733389] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1416.876918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1422.485718] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1430.007209] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 1430.009452] Lustre: Skipped 1 previous similar message [ 1432.034720] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.040115] Lustre: Skipped 2 previous similar messages [ 1440.240456] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 1440.243106] Lustre: Skipped 7 previous similar messages [ 1443.295136] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1443.347879] Lustre: server umount lustre-MDT0001 complete [ 1445.360059] LustreError: 101022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.557024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.587929] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1446.590102] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1446.602481] Lustre: 100296:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1446.830848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1446.833371] Lustre: Skipped 3 previous similar messages [ 1447.910868] LustreError: 104245:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1447.913284] LustreError: 104245:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1447.915905] LustreError: 100300:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1447.944637] LustreError: 104253:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.203.123@tcp: cfg command failed: rc = -17 [ 1447.947320] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1447.947320] [ 1448.201343] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1451.297414] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 18:31:45 (1778279505) [ 1452.007396] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1456.613437] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1462.467262] Lustre: 106149:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1470.880580] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1472.109937] Lustre: 107283:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1523.683260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1523.683688] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.685348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.685353] Lustre: Skipped 3 previous similar messages [ 1523.700211] Lustre: Skipped 3 previous similar messages [ 1528.941088] Lustre: server umount lustre-MDT0000 complete [ 1530.739819] LustreError: 101416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778279585 with bad export cookie 6409679532577788353 [ 1530.743588] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.745669] LustreError: 101416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1530.915882] Lustre: server umount lustre-MDT0001 complete [ 1543.197750] Lustre: server umount lustre-OST0000 complete [ 1555.108131] Lustre: server umount lustre-OST0001 complete [ 1557.149786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1560.453187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1563.278450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1566.822658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1573.917647] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1578.110839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.120971] Lustre: lustre-MDT0000: reset Object Index mappings [ 1578.311355] LustreError: 111186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1578.351307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.924373] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1583.425899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.436691] Lustre: lustre-MDT0001: reset Object Index mappings [ 1585.283671] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1586.414651] Lustre: 112300:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1589.099628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1591.729394] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1593.511933] LustreError: 112656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1593.519045] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1594.970281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1596.914993] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1597.319204] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1604.079257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1604.079843] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1611.353264] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1612.718186] Lustre: 114167:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1635.400723] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:34:49 (1778279689) [ 1665.508257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1665.509113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.518163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1665.520188] Lustre: Skipped 10 previous similar messages [ 1670.726896] Lustre: server umount lustre-MDT0000 complete [ 1672.235552] LustreError: 111168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778279726 with bad export cookie 6409679532577835120 [ 1672.239366] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1672.240261] LustreError: 111168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1675.743945] LustreError: 111218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1675.748007] LustreError: 111218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1678.513713] Lustre: server umount lustre-MDT0001 complete [ 1694.175092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1694.206890] Lustre: server umount lustre-OST0000 complete [ 1701.596054] Lustre: server umount lustre-OST0001 complete [ 1703.630964] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 1706.755167] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1718.683074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.773732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1718.784908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1718.820496] Lustre: lustre-MDT0000: new disk, initializing [ 1718.844987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.847102] Lustre: Skipped 3 previous similar messages [ 1718.852640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1720.166800] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1723.101144] Lustre: server umount lustre-MDT0000 complete [ 1726.261914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.357931] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1726.363134] Lustre: 118375:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1726.370799] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1727.710753] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1729.397714] Lustre: server umount lustre-MDT0000 complete [ 1732.060312] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:36:26 (1778279786) [ 1732.516636] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1733.009647] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:36:27 (1778279787) [ 1738.286283] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 1741.742588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1743.151018] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1746.099518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.141137] Lustre: 120391:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1746.168224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1746.170727] Lustre: Skipped 1 previous similar message [ 1746.210758] Lustre: lustre-MDT0001: new disk, initializing [ 1746.268766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1746.272520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1747.676296] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1749.965229] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1752.153883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1752.253696] Lustre: lustre-OST0000: new disk, initializing [ 1752.256530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1752.781733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1752.785544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1752.818921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1754.190119] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1758.299926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1759.747317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1759.752347] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1759.788442] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1760.356353] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 1764.955657] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1766.167853] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1878.940597] Lustre: 129142:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1878.945119] Lustre: 129142:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1878.948874] Lustre: 129142:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1878.951792] Lustre: 129142:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1878.954048] Lustre: 129142:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1878.956941] Lustre: 129142:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1956.170098] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:40:10 (1778280010) [ 1969.634322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.639690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.639920] Lustre: Skipped 8 previous similar messages [ 1969.642978] Lustre: Skipped 10 previous similar messages [ 1973.883157] Lustre: server umount lustre-MDT0000 complete [ 1974.752105] LustreError: 123585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.759055] LustreError: 123585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1975.083187] LustreError: 120381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778280029 with bad export cookie 6409679532577889895 [ 1975.086932] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1975.088902] LustreError: 120381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1981.330760] Lustre: server umount lustre-MDT0001 complete [ 1988.733648] Lustre: server umount lustre-OST0000 complete [ 1995.990052] Lustre: server umount lustre-OST0001 complete [ 1997.993089] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2001.391096] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2013.578488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.686776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2013.700777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2013.703498] Lustre: Skipped 1 previous similar message [ 2013.742873] Lustre: lustre-MDT0000: new disk, initializing [ 2013.744253] Lustre: Skipped 1 previous similar message [ 2013.768803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.771773] Lustre: Skipped 5 previous similar messages [ 2013.779273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2015.178903] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2019.276806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.317676] Lustre: 132120:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2019.415730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2020.692264] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2023.828962] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2025.029884] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2027.267905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2029.311825] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2029.367958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2029.411093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2032.524988] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2035.734297] Lustre: lustre-MDD0000: changelog on [ 2036.643387] Lustre: lustre-MDD0001: changelog on [ 2350.301970] Lustre: 132129:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2378.062686] Lustre: lustre-MDD0001: changelog off [ 2379.229748] Lustre: lustre-MDD0000: changelog off [ 2382.817613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2382.817948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.820106] LustreError: Skipped 1 previous similar message [ 2382.823342] Lustre: Skipped 2 previous similar messages [ 2382.824072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.830926] Lustre: Skipped 5 previous similar messages [ 2395.103642] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2395.165145] Lustre: server umount lustre-OST0000 complete [ 2398.178543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2402.616539] Lustre: server umount lustre-MDT0000 complete [ 2403.296109] LustreError: 132133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.304399] LustreError: 132133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2403.897390] LustreError: 132114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778280458 with bad export cookie 6409679532578691920 [ 2403.900052] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2403.903116] LustreError: 132114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2404.043597] Lustre: server umount lustre-MDT0001 complete [ 2407.656261] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2408.609558] Key type lgssc unregistered [ 2408.721328] LNet: 142358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2408.723779] LNetError: 142358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2408.731295] LNet: Removed LNI 192.168.203.123@tcp [ 2409.034099] Key type .llcrypt unregistered [ 2409.035773] Key type ._llcrypt unregistered [ 2415.281151] Key type ._llcrypt registered [ 2415.282468] Key type .llcrypt registered [ 2415.321891] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2421.450065] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2421.798893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2421.840045] alg: No test for adler32 (adler32-zlib) [ 2422.687411] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 2422.768135] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2424.343115] Key type lgssc registered [ 2424.699623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2435.959277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2435.964716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.047389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2437.060877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2437.103729] Lustre: lustre-MDT0000: new disk, initializing [ 2437.129297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.135929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2438.448835] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2442.546795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.586498] Lustre: 145495:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2442.609925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2442.611788] Lustre: Skipped 1 previous similar message [ 2442.649831] Lustre: lustre-MDT0001: new disk, initializing [ 2442.680477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2442.691487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2442.694467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2444.027026] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2447.151302] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2448.384991] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2450.630400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2450.730696] Lustre: lustre-OST0000: new disk, initializing [ 2450.732886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2450.759372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.088495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2452.091464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2452.124525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2452.613746] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2455.637290] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2457.942968] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_module obdecho/obdecho [ 2458.572963] LustreError: 148109:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2458.575747] LustreError: 148109:0:(obd_config.c:843:class_setup()) setup ec failed (-22) [ 2459.181644] LustreError: 148205:0:(echo_client.c:1881:echo_md_handler()) Can not resolve the path : rc = -2 [ 2462.688831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2462.690680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2462.694497] Lustre: Skipped 2 previous similar messages [ 2462.699086] Lustre: Skipped 2 previous similar messages [ 2467.810212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2467.812059] Lustre: Skipped 1 previous similar message [ 2468.570244] Lustre: server umount lustre-MDT0000 complete [ 2469.792401] LustreError: 145488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778280524 with bad export cookie 16392064192382496417 [ 2469.795756] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2469.797930] LustreError: 145488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2469.900848] Lustre: server umount lustre-MDT0001 complete [ 2481.375171] Lustre: server umount lustre-OST0000 complete [ 2488.036955] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2491.481578] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2502.910987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.993917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2503.004873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2503.040594] Lustre: lustre-MDT0000: new disk, initializing [ 2503.065904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2503.073829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2504.350735] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2508.549201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.599629] Lustre: 151881:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2508.645478] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2508.647251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2508.649434] Lustre: Skipped 1 previous similar message [ 2508.685500] Lustre: lustre-MDT0001: new disk, initializing [ 2508.723225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2508.736624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2508.741459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2510.074328] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2513.085203] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2514.264263] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2516.536440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.497774] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2521.562035] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2522.611068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2522.616112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2522.649095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2522.898265] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2522.968474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2524.282925] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2524.349374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2527.713308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2527.713891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.715830] LustreError: Skipped 1 previous similar message [ 2527.720462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2527.723706] Lustre: Skipped 1 previous similar message [ 2531.085495] Lustre: server umount lustre-OST0000 complete [ 2532.832196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2532.832776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2532.837041] Lustre: Skipped 2 previous similar messages [ 2538.459158] Lustre: server umount lustre-MDT0000 complete [ 2539.688049] LustreError: 152659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778280594 with bad export cookie 16392064192382497362 [ 2539.690905] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.693085] LustreError: 152659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2539.809493] Lustre: server umount lustre-MDT0001 complete [ 2541.004618] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:49:55 (1778280595) [ 2541.543631] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2542.158204] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:49:56 (1778280596) [ 2544.665679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.829931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.833105] Lustre: Skipped 1 previous similar message [ 2546.088037] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2548.815652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.269197] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2552.120612] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2553.349356] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2555.682485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.731812] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2559.566272] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2577.037360] LustreError: 158146:0:(ldlm_resource.c:1170:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd1:0x3b:0x0].0x0 (0000000077dba604) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2579.935905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2579.939795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.946400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2579.949388] Lustre: Skipped 3 previous similar messages [ 2583.194769] Lustre: server umount lustre-MDT0000 complete [ 2584.382441] LustreError: 155864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778280639 with bad export cookie 16392064192382498244 [ 2584.384802] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2584.386776] LustreError: 155864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2585.583904] LustreError: 155878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.589804] LustreError: 155878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2586.591792] LustreError: 155876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.592041] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.600369] Lustre: Skipped 2 previous similar messages [ 2589.600105] LustreError: 155877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.595578] Lustre: server umount lustre-MDT0001 complete [ 2592.666462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2596.851234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.928891] LustreError: 159406:0:(llog.c:1646:llog_backup()) MGC192.168.203.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2596.931675] Lustre: 159406:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2606.063675] LustreError: 159410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.183991] LustreError: 159410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.189197] LustreError: 159410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2619.359807] LustreError: 159411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.365892] LustreError: 159411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2636.785188] LustreError: 159430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.791147] LustreError: 159430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 2657.887192] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2657.888970] Lustre: 159406:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.203.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2670.560126] LustreError: 159410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.564994] LustreError: 159410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 2698.727282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2698.729664] Lustre: Skipped 2 previous similar messages [ 2699.931554] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2702.792335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.843069] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2704.863157] LustreError: 160130:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2704.863526] LustreError: 143724:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000059054b72 x1864662229549056/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2704.875661] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe37c4f88ba5c0eec [ 2704.878439] Lustre: MGC192.168.203.123@tcp: Connection restored to 0@lo (at 0@lo) [ 2706.189909] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2707.827993] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2708.305100] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2709.390253] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2709.990670] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2714.003297] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec [ 2720.225926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.226849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2720.229780] Lustre: Skipped 2 previous similar messages [ 2720.231613] Lustre: Skipped 8 previous similar messages [ 2720.233182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2720.338410] Lustre: server umount lustre-MDT0000 complete [ 2721.569180] LustreError: 155863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778280776 with bad export cookie 16392064192382504684 [ 2721.573470] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2721.574300] LustreError: 155863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2721.697525] Lustre: server umount lustre-MDT0001 complete [ 2733.209802] Lustre: server umount lustre-OST0000 complete [ 2739.687952] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2743.017328] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2754.762427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.852587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2754.864171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2754.866278] Lustre: Skipped 1 previous similar message [ 2754.902480] Lustre: lustre-MDT0000: new disk, initializing [ 2754.903847] Lustre: Skipped 1 previous similar message [ 2754.926185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.928164] Lustre: Skipped 1 previous similar message [ 2754.933157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2756.263980] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2760.321118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.357549] Lustre: 164221:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2760.447576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2761.683142] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2764.560848] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2765.616793] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2767.634047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2769.211354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2769.247181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2769.523365] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2772.414316] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2773.639602] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2773.705107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2774.953943] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2775.015866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2779.617719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.618618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2779.618847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2779.623126] Lustre: Skipped 1 previous similar message [ 2781.726442] Lustre: server umount lustre-OST0000 complete [ 2789.081071] Lustre: server umount lustre-MDT0000 complete [ 2789.855921] LustreError: 165014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2789.860203] LustreError: 165014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 2790.215937] LustreError: 164214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778280844 with bad export cookie 16392064192382521484 [ 2790.219190] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2790.220958] LustreError: 164214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2790.330970] Lustre: server umount lustre-MDT0001 complete [ 2793.085480] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:54:07 (1778280847) [ 2795.089694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.392178] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2798.946367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.198368] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2801.760549] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2802.834748] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2804.843241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2806.735618] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2808.325968] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2816.251788] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:54:30 (1778280870) [ 2819.557263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2819.557796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.559713] LustreError: Skipped 1 previous similar message [ 2819.563043] Lustre: Skipped 2 previous similar messages [ 2823.313917] Lustre: server umount lustre-OST0000 complete [ 2830.616411] Lustre: server umount lustre-MDT0000 complete [ 2831.893173] LustreError: 169838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778280886 with bad export cookie 16392064192382522352 [ 2831.895965] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2831.896404] LustreError: 169838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2832.024115] Lustre: server umount lustre-MDT0001 complete [ 2835.477867] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2836.478118] Key type lgssc unregistered [ 2836.600325] LNet: 171592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2836.602956] LNetError: 171592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2836.611316] LNet: Removed LNI 192.168.203.123@tcp [ 2836.893933] Key type .llcrypt unregistered [ 2836.895108] Key type ._llcrypt unregistered [ 2844.516793] Key type ._llcrypt registered [ 2844.518060] Key type .llcrypt registered [ 2844.555323] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2844.898638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2844.939142] alg: No test for adler32 (adler32-zlib) [ 2845.791295] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 2845.865662] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 2847.447119] Key type lgssc registered [ 2847.799789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2850.319520] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2850.325289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2865.759256] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2870.879116] LustreError: 172794:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.123@tcp: failed processing log, type 4: rc = -110 [ 2875.999334] LustreError: 172794:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2876.005430] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2876.007224] LustreError: MGC192.168.203.123@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2876.012824] LustreError: 172794:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2876.015931] LustreError: 172794:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -22 [ 2876.019070] LustreError: 172794:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 2876.041789] Lustre: server umount lustre-OST0000 complete [ 2876.043513] LustreError: 172794:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2878.835571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2879.991875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2881.246318] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2884.090352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2884.208491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2885.579298] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2887.337486] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2888.510087] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2890.803448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2890.890475] LustreError: 174881:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2890.896518] Lustre: 174881:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.203.123@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2890.948817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2892.759812] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2894.546471] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2899.430735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2899.433546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2899.433942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2899.439071] Lustre: Skipped 2 previous similar messages [ 2899.443593] Lustre: Skipped 1 previous similar message [ 2901.736725] Lustre: server umount lustre-MDT0000 complete [ 2903.325339] LustreError: 173272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778280958 with bad export cookie 3674530525065320027 [ 2903.329471] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2903.330942] LustreError: 173272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2903.460834] Lustre: server umount lustre-MDT0001 complete [ 2915.380999] Lustre: server umount lustre-OST0000 complete [ 2919.346097] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 2923.980175] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 2940.325676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.411049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2940.422459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2940.469560] Lustre: lustre-MDT0000: new disk, initializing [ 2940.492799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.498738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2941.898327] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2947.065711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.127193] Lustre: 178251:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2947.155225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2947.158445] Lustre: Skipped 1 previous similar message [ 2947.199530] Lustre: lustre-MDT0001: new disk, initializing [ 2947.243464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2947.259638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2947.264710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2948.939277] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2952.624080] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2954.121075] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2957.264409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2957.425601] Lustre: lustre-OST0000: new disk, initializing [ 2957.428653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2957.468347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2959.815237] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 2962.441908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2962.448100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2962.493642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2963.491655] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2964.959239] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2965.032591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2966.702617] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2966.787732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2972.645173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2972.645315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.650432] LustreError: Skipped 1 previous similar message [ 2972.658442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2972.662949] Lustre: Skipped 1 previous similar message [ 2973.724993] Lustre: server umount lustre-OST0000 complete [ 2977.761106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.761658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2977.765290] Lustre: Skipped 2 previous similar messages [ 2977.769810] Lustre: Skipped 1 previous similar message [ 2981.554979] Lustre: server umount lustre-MDT0000 complete [ 2982.880891] LustreError: 179682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2982.892445] LustreError: 179682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2983.281686] LustreError: 178245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778281037 with bad export cookie 3674530525065320713 [ 2983.286339] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2983.289714] LustreError: 178245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2983.450816] Lustre: server umount lustre-MDT0001 complete [ 2987.253551] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:57:21 (1778281041) [ 2995.081254] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 2996.495371] Key type lgssc unregistered [ 2996.664556] LNet: 182768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2996.669396] LNetError: 182768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2996.679533] LNet: Removed LNI 192.168.203.123@tcp [ 2997.171310] Key type .llcrypt unregistered [ 2997.173581] Key type ._llcrypt unregistered [ 3005.568398] Key type ._llcrypt registered [ 3005.570309] Key type .llcrypt registered [ 3005.609720] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3005.939642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3005.946803] alg: No test for adler32 (adler32-zlib) [ 3006.816854] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 3006.918505] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3008.519176] Key type lgssc registered [ 3008.917874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3012.725071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3012.739103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.945228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.689370] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3019.962995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.126023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3021.899639] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3024.477486] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3025.991205] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3029.864842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3029.990145] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3029.993122] LustreError: 185584:0:(llog.c:1687:llog_backup()) MGC192.168.203.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3029.999343] Lustre: 185584:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.203.123@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3030.066485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.641577] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3035.195178] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3036.243883] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.246126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.252332] Lustre: Skipped 2 previous similar messages [ 3036.256357] Lustre: Skipped 1 previous similar message [ 3041.249792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.253975] Lustre: Skipped 3 previous similar messages [ 3042.414985] Lustre: server umount lustre-MDT0000 complete [ 3044.143909] LustreError: 183927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778281098 with bad export cookie 18274707503278860951 [ 3044.147386] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3044.152192] LustreError: 183927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3044.307833] Lustre: server umount lustre-MDT0001 complete [ 3056.219899] Lustre: server umount lustre-OST0000 complete [ 3059.123339] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3062.882668] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3074.685192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.775940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3074.787872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3074.828467] Lustre: lustre-MDT0000: new disk, initializing [ 3074.853839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3074.860549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3076.176507] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3080.315462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.350989] Lustre: 188955:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3080.368213] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3080.369464] Lustre: Skipped 1 previous similar message [ 3080.370657] LustreError: 189719:0:(llog.c:1687:llog_backup()) MGC192.168.203.123@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3080.373255] LustreError: 189719:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3080.375161] Lustre: 189719:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.123@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3080.382204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3080.383959] Lustre: Skipped 1 previous similar message [ 3080.423120] Lustre: lustre-MDT0001: new disk, initializing [ 3080.453417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3080.463495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3080.466823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3081.668606] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3084.707578] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3085.920917] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3088.288078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3088.374850] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3088.376013] Lustre: Skipped 1 previous similar message [ 3088.377265] LustreError: 190673:0:(llog.c:1687:llog_backup()) MGC192.168.203.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3088.379896] LustreError: 190673:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3088.382695] Lustre: 190673:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.123@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3088.386861] Lustre: 190673:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3088.394759] Lustre: lustre-OST0000: new disk, initializing [ 3088.396494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3090.210614] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3090.553346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3090.556341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3090.587839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3093.408157] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3094.909516] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3094.990561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3096.266469] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3096.336861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3101.154285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3101.154699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.156816] LustreError: Skipped 1 previous similar message [ 3101.163279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3101.165978] Lustre: Skipped 1 previous similar message [ 3106.274823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3106.278086] Lustre: Skipped 1 previous similar message [ 3111.391145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3111.394572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3111.396509] Lustre: Skipped 1 previous similar message [ 3111.447621] Lustre: server umount lustre-OST0000 complete [ 3116.514413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.514620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3116.517963] Lustre: Skipped 2 previous similar messages [ 3118.809632] Lustre: server umount lustre-MDT0000 complete [ 3120.117248] LustreError: 188948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778281174 with bad export cookie 18274707503278861630 [ 3120.120614] LustreError: 188948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3120.120842] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3120.237592] Lustre: server umount lustre-MDT0001 complete [ 3123.068584] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:59:37 (1778281177) [ 3128.733500] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3132.946170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3133.097050] LustreError: 193657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.102474] LustreError: 193657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3133.113516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.117087] Lustre: Skipped 1 previous similar message [ 3134.539427] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3137.926289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.521794] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3140.567356] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3143.281388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3146.143375] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3149.023461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.093222] Lustre: lustre-OST0001: new disk, initializing [ 3149.095705] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3149.133073] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3149.135265] Lustre: Skipped 2 previous similar messages [ 3150.022356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3150.025949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3150.073298] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3151.361156] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3156.850883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3164.689431] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3170.500454] Lustre: DEBUG MARKER: ADD OST3 [ 3172.483591] loop: module loaded [ 3172.499877] loop0: detected capacity change from 0 to 204800000 [ 3172.599856] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3172.603294] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3172.606946] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3174.930158] loop0: detected capacity change from 0 to 204800000 [ 3177.325608] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3177.395495] Lustre: lustre-OST0003: new disk, initializing [ 3177.397901] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3177.569973] Lustre: DEBUG MARKER: STOP OST3 [ 3178.216308] Lustre: Failing over lustre-OST0003 [ 3178.246731] Lustre: server umount lustre-OST0003 complete [ 3178.496189] LustreError: 195109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3179.408719] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3180.528812] LustreError: 195109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3180.536840] LustreError: 195109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3182.962979] Lustre: DEBUG MARKER: START OST3 again [ 3183.535994] LustreError: 197968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3185.319156] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.433933] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3185.437220] Lustre: Skipped 1 previous similar message [ 3186.869127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3186.874229] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3186.885206] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3187.690126] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3201.799259] Lustre: Failing over lustre-OST0003 [ 3201.853889] Lustre: server umount lustre-OST0003 complete [ 3202.528128] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3202.528275] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3202.532067] LustreError: Skipped 1 previous similar message [ 3202.534394] LustreError: 195108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3202.535508] Lustre: Skipped 1 previous similar message [ 3202.542989] LustreError: 195108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3214.311072] LustreError: 195642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3214.322767] LustreError: 195642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3214.815611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3214.818220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.823230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3214.825419] Lustre: Skipped 2 previous similar messages [ 3221.166140] Lustre: server umount lustre-MDT0000 complete [ 3222.582059] LustreError: 193637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778281277 with bad export cookie 18274707503278862491 [ 3222.586306] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3222.588625] LustreError: 193637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3224.545982] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.553846] Lustre: Skipped 3 previous similar messages [ 3228.816701] Lustre: server umount lustre-MDT0001 complete [ 3230.287780] Lustre: server umount lustre-OST0000 complete [ 3231.552199] Lustre: server umount lustre-OST0001 complete [ 3233.388362] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3235.853969] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3253.023558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.131109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3253.148933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3253.202612] Lustre: lustre-MDT0000: new disk, initializing [ 3253.246321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.257926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3255.066220] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3260.411651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.470526] Lustre: 202991:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3260.622581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3262.445570] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3266.361817] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3268.063635] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3271.111498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3271.239426] Lustre: lustre-OST0000: new disk, initializing [ 3271.241503] Lustre: Skipped 1 previous similar message [ 3271.244072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3271.246783] Lustre: Skipped 2 previous similar messages [ 3272.338497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3272.343243] Lustre: Skipped 1 previous similar message [ 3272.346546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3272.398889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3273.814825] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3277.540890] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3279.255394] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3279.348578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3280.951079] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3281.046922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3282.913442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.914127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3282.915217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3282.915222] Lustre: Skipped 8 previous similar messages [ 3282.919938] Lustre: Skipped 2 previous similar messages [ 3296.223086] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3296.285940] Lustre: server umount lustre-OST0000 complete [ 3303.711416] Lustre: server umount lustre-MDT0000 complete [ 3304.958373] LustreError: 202981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778281359 with bad export cookie 18274707503278864731 [ 3304.962183] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3304.963295] LustreError: 202981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3305.084053] Lustre: server umount lustre-MDT0001 complete [ 3306.272328] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:02:40 (1778281360) [ 3312.127928] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3315.650856] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3329.181087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.269877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3329.284476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3329.323970] Lustre: lustre-MDT0000: new disk, initializing [ 3329.356382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3330.844056] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3335.184239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.222789] Lustre: 209149:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3335.311762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3336.552560] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3339.655037] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3340.912704] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3343.223874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3344.476055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3345.160310] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3348.317617] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3349.624779] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3349.691534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3350.946181] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3351.011412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3354.593501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3354.595881] LustreError: Skipped 1 previous similar message [ 3354.596236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.597566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3354.601064] Lustre: Skipped 3 previous similar messages [ 3354.604160] Lustre: Skipped 10 previous similar messages [ 3357.711494] Lustre: server umount lustre-OST0000 complete [ 3365.078382] Lustre: server umount lustre-MDT0000 complete [ 3366.321074] LustreError: 209941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778281421 with bad export cookie 18274707503278865613 [ 3366.323640] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3366.324957] LustreError: 209941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3366.425821] Lustre: server umount lustre-MDT0001 complete [ 3369.095231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.237037] LustreError: 212956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3369.244888] LustreError: 212956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3370.524193] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3373.404677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.160207] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3377.408564] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3378.880924] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3381.401583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3381.557138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.559971] Lustre: Skipped 7 previous similar messages [ 3383.843434] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3386.029291] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3404.259205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3404.262572] LustreError: Skipped 1 previous similar message [ 3409.561415] Lustre: server umount lustre-OST0000 complete [ 3417.244383] Lustre: server umount lustre-MDT0000 complete [ 3418.893157] LustreError: 215517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778281473 with bad export cookie 18274707503278866495 [ 3418.898095] LustreError: 215517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3418.898318] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3419.047783] Lustre: server umount lustre-MDT0001 complete [ 3519.759624] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 3520.823828] Key type lgssc unregistered [ 3520.969715] LNet: 216287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3520.974335] LNetError: 216287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3520.985518] LNet: Removed LNI 192.168.203.123@tcp [ 3521.306106] Key type .llcrypt unregistered [ 3521.307592] Key type ._llcrypt unregistered [ 3531.140429] Key type ._llcrypt registered [ 3531.141867] Key type .llcrypt registered [ 3531.184976] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3538.163984] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3538.529543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3538.562797] alg: No test for adler32 (adler32-zlib) [ 3539.445207] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 3539.538959] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 3541.127154] Key type lgssc registered [ 3541.588667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3555.874222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3555.884817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.001050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3557.021654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3557.096737] Lustre: lustre-MDT0000: new disk, initializing [ 3557.151774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3557.161655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3558.914865] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3564.404771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.452586] Lustre: 219578:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3564.486471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3564.489725] Lustre: Skipped 1 previous similar message [ 3564.544387] Lustre: lustre-MDT0001: new disk, initializing [ 3564.596570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3564.619677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3564.623675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3566.219277] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3569.811340] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3571.357062] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3574.173652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3574.298829] Lustre: lustre-OST0000: new disk, initializing [ 3574.302107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3574.337844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.633572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3574.638384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3574.687995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3576.883939] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3581.000600] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3582.754763] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3582.848527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3584.587122] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3584.677818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3590.113463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3590.115526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3590.121944] LustreError: Skipped 1 previous similar message [ 3590.123787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3590.129083] Lustre: Skipped 1 previous similar message [ 3590.142376] Lustre: Skipped 1 previous similar message [ 3591.656655] Lustre: server umount lustre-OST0000 complete [ 3595.232781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3595.234228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.238505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3595.243279] Lustre: Skipped 1 previous similar message [ 3595.249046] Lustre: Skipped 1 previous similar message [ 3599.621332] Lustre: server umount lustre-MDT0000 complete [ 3601.380760] LustreError: 219567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778281656 with bad export cookie 2234595812448947858 [ 3601.384101] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3601.387673] LustreError: 219567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3601.514028] Lustre: server umount lustre-MDT0001 complete [ 3609.157215] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3613.591911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3613.792210] LustreError: 224081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3613.802193] LustreError: 224081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3613.817378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.382793] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3618.829336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.999545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3620.586695] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3621.769186] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3624.443786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3626.937817] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3630.536378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.610202] Lustre: lustre-OST0001: new disk, initializing [ 3630.612430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3630.649964] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3630.653011] Lustre: Skipped 1 previous similar message [ 3632.879798] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3637.253274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3637.258377] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3637.286201] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3642.967355] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3651.135865] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3660.525756] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:08:34 (1778281714) [ 3663.842816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.844187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3663.849023] Lustre: Skipped 2 previous similar messages [ 3663.851073] Lustre: Skipped 1 previous similar message [ 3667.426291] Lustre: server umount lustre-MDT0000 complete [ 3668.786210] LustreError: 224061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778281723 with bad export cookie 2234595812448948726 [ 3668.789724] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3668.792837] LustreError: 224061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3668.924440] Lustre: server umount lustre-MDT0001 complete [ 3680.583557] Lustre: server umount lustre-OST0000 complete [ 3691.433958] Lustre: server umount lustre-OST0001 complete [ 3693.679865] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 3697.217167] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 3709.834366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.921698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3709.933724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3709.977561] Lustre: lustre-MDT0000: new disk, initializing [ 3710.005356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3710.013373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3711.442156] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3715.826239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3715.864158] Lustre: 230231:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3717.261312] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3720.475535] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3721.806216] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3724.234108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3724.337915] Lustre: lustre-OST0000: new disk, initializing [ 3724.339311] Lustre: Skipped 1 previous similar message [ 3724.341532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3724.345295] Lustre: Skipped 2 previous similar messages [ 3726.005554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3726.008314] Lustre: Skipped 1 previous similar message [ 3726.010892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3726.045321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3726.317648] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3729.513552] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3730.980125] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3731.056416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3732.438666] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3732.505277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3736.545619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3736.546533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.548783] LustreError: Skipped 1 previous similar message [ 3736.549556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.554871] Lustre: Skipped 2 previous similar messages [ 3739.226248] Lustre: server umount lustre-OST0000 complete [ 3741.665142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3741.665755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.666492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3741.666497] Lustre: Skipped 1 previous similar message [ 3741.680526] Lustre: Skipped 1 previous similar message [ 3746.715567] Lustre: server umount lustre-MDT0000 complete [ 3746.785396] LustreError: 230243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3746.794816] LustreError: 230243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3748.136298] LustreError: 231974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778281802 with bad export cookie 2234595812448950847 [ 3748.139695] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3748.142058] LustreError: 231974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3748.261772] Lustre: server umount lustre-MDT0001 complete [ 3751.521042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3751.719333] LustreError: 234044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.758331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3751.760992] Lustre: Skipped 2 previous similar messages [ 3753.298188] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3756.400263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.967675] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3759.939956] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3761.235792] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3763.691749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3765.886267] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 3767.767868] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3910.625734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3910.626127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.627374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3910.627379] Lustre: Skipped 1 previous similar message [ 3910.629900] LustreError: Skipped 1 previous similar message [ 3910.640865] Lustre: Skipped 1 previous similar message [ 3916.692617] Lustre: server umount lustre-OST0000 complete [ 3920.352717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3924.388868] Lustre: server umount lustre-MDT0000 complete [ 3925.884282] LustreError: 235657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778281980 with bad export cookie 2234595812448951736 [ 3925.887553] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3925.889271] LustreError: 235657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3926.002434] Lustre: server umount lustre-MDT0001 complete [ 4032.330387] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4033.434300] Key type lgssc unregistered [ 4033.568341] LNet: 237294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4033.571821] LNetError: 237294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4033.580367] LNet: Removed LNI 192.168.203.123@tcp [ 4033.916156] Key type .llcrypt unregistered [ 4033.917384] Key type ._llcrypt unregistered [ 4041.562466] Key type ._llcrypt registered [ 4041.563708] Key type .llcrypt registered [ 4041.594580] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4047.573201] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4047.901362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4047.948067] alg: No test for adler32 (adler32-zlib) [ 4048.792741] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 4048.873170] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4050.455149] Key type lgssc registered [ 4050.842279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4064.081464] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4067.378802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4067.384730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.473755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4068.485526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4068.520963] Lustre: lustre-MDT0000: new disk, initializing [ 4068.540725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.547230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4069.852069] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4073.820666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.860693] Lustre: 241285:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4073.883205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4073.885140] Lustre: Skipped 1 previous similar message [ 4073.920335] Lustre: lustre-MDT0001: new disk, initializing [ 4073.947643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4073.957107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4073.960692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4075.139491] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4077.356090] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4079.460148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4079.547404] Lustre: lustre-OST0000: new disk, initializing [ 4079.549208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4079.575615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4081.343593] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4082.357045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4082.360048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4082.393673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4085.283388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4085.337748] Lustre: lustre-OST0001: new disk, initializing [ 4085.339401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4085.368895] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4087.117996] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4089.853210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4089.856694] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4089.883838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4092.394057] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4099.012328] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4109.792191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4109.794843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.798625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4110.304341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4110.306465] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.923213] Lustre: server umount lustre-MDT0000 complete [ 4114.102064] LustreError: 241276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778282168 with bad export cookie 5056750009426257185 [ 4114.104722] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4114.106513] LustreError: 241276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4114.223193] Lustre: server umount lustre-MDT0001 complete [ 4125.699511] Lustre: server umount lustre-OST0000 complete [ 4137.130527] Lustre: server umount lustre-OST0001 complete [ 4141.840390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4145.034315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4147.561636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4150.374584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4156.017686] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4159.321354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.329708] Lustre: lustre-MDT0000: reset Object Index mappings [ 4159.490591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.760803] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4163.398743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.406044] Lustre: lustre-MDT0001: reset Object Index mappings [ 4164.695555] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4165.574878] Lustre: 249815:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4167.672023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4167.800081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4167.802240] Lustre: Skipped 1 previous similar message [ 4169.645474] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4169.826317] LustreError: 250169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4169.830348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4169.831667] LustreError: 250169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4172.324907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4173.417113] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4174.252204] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4177.289690] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4178.409550] Lustre: 251656:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4189.154577] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.156692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4189.159166] Lustre: Skipped 5 previous similar messages [ 4189.161674] Lustre: Skipped 5 previous similar messages [ 4189.164377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4190.622322] Lustre: server umount lustre-MDT0000 complete [ 4191.786132] LustreError: 248688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778282246 with bad export cookie 5056750009426263961 [ 4191.788896] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4191.789776] LustreError: 248688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4191.903926] Lustre: server umount lustre-MDT0001 complete [ 4202.435891] Lustre: server umount lustre-OST0000 complete [ 4213.916243] Lustre: server umount lustre-OST0001 complete [ 4220.159686] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:17:54 (1778282274) [ 4225.367409] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4228.660145] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4239.970662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4240.051929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4240.063327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4240.097401] Lustre: lustre-MDT0000: new disk, initializing [ 4240.120878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4240.122662] Lustre: Skipped 1 previous similar message [ 4240.127310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4241.404820] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4245.495554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4245.534537] Lustre: 255869:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4245.632143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4246.940261] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4250.016189] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4251.215689] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4253.386636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4253.489901] Lustre: lustre-OST0000: new disk, initializing [ 4253.491510] Lustre: Skipped 1 previous similar message [ 4253.493333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4253.495691] Lustre: Skipped 2 previous similar messages [ 4255.097753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4255.101151] Lustre: Skipped 1 previous similar message [ 4255.103970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4255.137039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4255.373422] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4258.438839] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4259.806077] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4259.878036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4261.065691] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4261.131965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4265.440929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4265.441455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.443349] LustreError: Skipped 1 previous similar message [ 4265.447607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4265.450906] Lustre: Skipped 2 previous similar messages [ 4267.852986] Lustre: server umount lustre-OST0000 complete [ 4270.560481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.561018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4270.563860] Lustre: Skipped 2 previous similar messages [ 4280.800317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4280.802408] Lustre: Skipped 4 previous similar messages [ 4283.359093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4283.414194] Lustre: server umount lustre-MDT0000 complete [ 4284.627973] LustreError: 255862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778282339 with bad export cookie 5056750009426266047 [ 4284.631283] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4284.631576] LustreError: 255862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4284.745610] Lustre: server umount lustre-MDT0001 complete [ 4290.732918] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4294.255763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.418312] LustreError: 260376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4294.435033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.437396] Lustre: Skipped 2 previous similar messages [ 4295.782054] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4298.680776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.168381] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4301.156919] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4303.407764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4305.495203] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4308.405970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4308.468738] Lustre: lustre-OST0001: new disk, initializing [ 4308.470993] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4310.347491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4310.352118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4310.389874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4310.450546] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4314.915213] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4316.169456] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4322.952763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4322.957699] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4325.860217] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.862448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4325.866451] Lustre: Skipped 2 previous similar messages [ 4325.872224] Lustre: Skipped 3 previous similar messages [ 4330.912530] Lustre: server umount lustre-MDT0000 complete [ 4330.978520] LustreError: 262010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4330.984549] LustreError: 262010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4332.137255] LustreError: 260356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778282386 with bad export cookie 5056750009426266936 [ 4332.140585] LustreError: 260356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4332.140764] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4332.249549] Lustre: server umount lustre-MDT0001 complete [ 4344.333908] Lustre: server umount lustre-OST0000 complete [ 4355.863460] Lustre: server umount lustre-OST0001 complete [ 4361.225069] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4364.842245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4365.002115] LustreError: 265652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4365.023470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.025949] Lustre: Skipped 3 previous similar messages [ 4366.400284] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4369.299610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4370.763364] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4371.715158] Lustre: 266758:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4373.892689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4375.073690] LustreError: 267305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.078616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4375.902785] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4378.890668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4380.015325] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4380.935205] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4384.118676] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4385.309877] Lustre: 268605:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4392.069722] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4392.074265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4395.211209] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:20:49 (1778282449) [ 4410.336079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4410.338538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4410.341929] Lustre: Skipped 1 previous similar message [ 4410.343650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4412.897422] Lustre: server umount lustre-MDT0000 complete [ 4414.126143] LustreError: 266761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778282468 with bad export cookie 5056750009426273243 [ 4414.129603] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4414.129849] LustreError: 266761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4415.968044] LustreError: 267929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4415.973374] LustreError: 267929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4420.269432] Lustre: server umount lustre-MDT0001 complete [ 4427.648499] Lustre: server umount lustre-OST0000 complete [ 4443.103174] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4443.170738] Lustre: server umount lustre-OST0001 complete [ 4445.172215] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4448.191838] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4466.399458] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4471.647850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.782249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4471.810806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4471.879558] Lustre: lustre-MDT0000: new disk, initializing [ 4471.932609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4473.920667] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4479.489811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.549868] Lustre: 272787:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4479.698398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4481.567748] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4486.841430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.607499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4488.960789] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4493.255275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4493.363849] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4493.367014] Lustre: Skipped 6 previous similar messages [ 4495.045417] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4495.425443] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4500.014882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4510.211605] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 19:22:44 (1778282564) [ 4515.808810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4515.809301] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.812325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4515.815652] Lustre: Skipped 8 previous similar messages [ 4515.817361] Lustre: Skipped 9 previous similar messages [ 4517.028338] Lustre: server umount lustre-MDT0000 complete [ 4518.228951] LustreError: 272779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778282572 with bad export cookie 5056750009426281111 [ 4518.232308] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4518.233920] LustreError: 272779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4518.356904] Lustre: server umount lustre-MDT0001 complete [ 4530.179142] Lustre: server umount lustre-OST0000 complete [ 4541.669976] Lustre: server umount lustre-OST0001 complete [ 4547.872441] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4551.363108] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4563.496802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.579692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4563.583077] Lustre: Skipped 2 previous similar messages [ 4563.593023] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4563.595119] Lustre: Skipped 4 previous similar messages [ 4563.633286] Lustre: lustre-MDT0000: new disk, initializing [ 4563.635432] Lustre: Skipped 3 previous similar messages [ 4563.674611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4563.678361] Lustre: Skipped 3 previous similar messages [ 4564.940044] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4569.035139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.076320] Lustre: 279817:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4569.170080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4569.174642] Lustre: Skipped 2 previous similar messages [ 4570.425592] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4573.549408] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4574.894370] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4577.343726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4578.799231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4579.585357] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4582.761159] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4583.489520] Lustre: Mounted lustre-client [ 4583.810045] LustreError: 282315:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff91756e9e9800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4583.835738] Lustre: Unmounted lustre-client [ 4588.514521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.521793] Lustre: Skipped 1 previous similar message [ 4589.203804] Lustre: server umount lustre-OST0000 complete [ 4596.637250] Lustre: server umount lustre-MDT0000 complete [ 4597.894925] LustreError: 279810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778282652 with bad export cookie 5056750009426282637 [ 4597.899181] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4597.899583] LustreError: 279810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4598.023976] Lustre: server umount lustre-MDT0001 complete [ 4602.039269] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4603.133546] Key type lgssc unregistered [ 4640.159388] LNet: 283593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4640.163310] LNetError: 283593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4715.941306] LNet: Removed LNI 192.168.203.123@tcp [ 4716.234131] Key type .llcrypt unregistered [ 4716.235624] Key type ._llcrypt unregistered [ 4722.286362] Key type ._llcrypt registered [ 4722.287539] Key type .llcrypt registered [ 4722.323896] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4728.939847] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4729.308829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4729.328753] alg: No test for adler32 (adler32-zlib) [ 4730.180808] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 4730.266829] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4731.855133] Key type lgssc registered [ 4732.265747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4742.187569] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:26:36 (1778282796) [ 4744.343189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4744.350547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.430427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4745.441191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4745.477040] Lustre: lustre-MDT0000: new disk, initializing [ 4745.502484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4745.509960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4746.786874] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4750.900854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.941128] Lustre: 286757:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4750.963433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4750.965581] Lustre: Skipped 1 previous similar message [ 4751.002792] Lustre: lustre-MDT0001: new disk, initializing [ 4751.032815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4751.045381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4751.051335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4752.384597] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4755.464508] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4756.650910] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4758.895587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4759.008314] Lustre: lustre-OST0000: new disk, initializing [ 4759.011533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4759.045646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4761.020113] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4761.910543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4761.914871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4761.945793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4764.095933] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4770.074623] Lustre: server umount lustre-OST0000 complete [ 4771.807817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4771.810445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4771.815083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4772.323297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4772.323746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.440014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4777.442541] Lustre: Skipped 2 previous similar messages [ 4781.564827] Lustre: server umount lustre-MDT0000 complete [ 4782.874515] LustreError: 286750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778282837 with bad export cookie 11879242118048242794 [ 4782.878891] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4782.879117] LustreError: 286750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4793.364595] Lustre: server umount lustre-MDT0001 complete [ 4797.302028] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4798.395072] Key type lgssc unregistered [ 4798.525703] LNet: 290127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4798.528764] LNetError: 290127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4798.535337] LNet: Removed LNI 192.168.203.123@tcp [ 4798.843769] Key type .llcrypt unregistered [ 4798.845545] Key type ._llcrypt unregistered [ 4808.686213] Key type ._llcrypt registered [ 4808.687436] Key type .llcrypt registered [ 4808.734543] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4809.129278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4809.170513] alg: No test for adler32 (adler32-zlib) [ 4810.042884] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 4810.147365] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4811.735162] Key type lgssc registered [ 4812.176685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4815.228736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4815.239397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.423564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.816665] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4820.901415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4821.029709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4822.408073] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4824.384034] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4825.764794] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4828.219835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4828.334273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4830.310375] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4832.074976] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4845.274900] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4845.841954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4847.608675] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 4849.362567] Lustre: server umount lustre-MDT0000 complete [ 4850.642630] LustreError: 291483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778282905 with bad export cookie 4493310598448107731 [ 4850.645942] LustreError: 291483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4850.646488] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4852.192042] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4852.192413] LustreError: 291498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.196302] Lustre: Skipped 1 previous similar message [ 4852.200479] LustreError: 291498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4852.202098] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4852.719777] LustreError: 291499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.200110] LustreError: 291498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4855.201049] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4855.210563] Lustre: Skipped 1 previous similar message [ 4856.811414] Lustre: server umount lustre-MDT0001 complete [ 4862.966244] Lustre: lustre-OST0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 4864.278686] Lustre: server umount lustre-OST0000 complete [ 4867.349106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4868.678284] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4869.604737] Lustre: 295259:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.123@tcp [ 4870.277634] Lustre: server umount MGS complete [ 4872.951764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4873.089323] LustreError: 295830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4873.108205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4874.325375] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4877.060543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4877.172889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4878.451468] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4881.294659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4883.213043] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4884.980715] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4885.483573] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4886.615561] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4887.215233] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4891.617818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.618031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4891.623887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4891.627901] Lustre: Skipped 1 previous similar message [ 4895.179675] Lustre: server umount lustre-OST0000 complete [ 4896.736275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4896.742193] Lustre: Skipped 2 previous similar messages [ 4901.856631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4901.859722] Lustre: Skipped 3 previous similar messages [ 4902.560350] Lustre: server umount lustre-MDT0000 complete [ 4903.946233] LustreError: 295811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778282958 with bad export cookie 4493310598448108557 [ 4903.948721] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4903.952108] LustreError: 295811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4904.065722] Lustre: server umount lustre-MDT0001 complete [ 4907.788664] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 4908.786509] Key type lgssc unregistered [ 4908.909529] LNet: 299006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4908.912642] LNetError: 299006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4908.923321] LNet: Removed LNI 192.168.203.123@tcp [ 4909.215091] Key type .llcrypt unregistered [ 4909.216414] Key type ._llcrypt unregistered [ 4915.139093] Key type ._llcrypt registered [ 4915.140409] Key type .llcrypt registered [ 4915.178337] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4921.293299] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 4921.725602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4921.732059] alg: No test for adler32 (adler32-zlib) [ 4922.591430] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 4922.671587] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 4924.247189] Key type lgssc registered [ 4924.606182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4936.300574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4936.306110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4937.397623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4937.409658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4937.443675] Lustre: lustre-MDT0000: new disk, initializing [ 4937.472517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4937.479709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4938.833336] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4943.186320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4943.223780] Lustre: 302100:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4943.247694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4943.249480] Lustre: Skipped 1 previous similar message [ 4943.286124] Lustre: lustre-MDT0001: new disk, initializing [ 4943.323325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4943.336781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4943.340406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4944.652623] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4947.723371] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4948.935574] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4951.291926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4951.397159] Lustre: lustre-OST0000: new disk, initializing [ 4951.399986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4951.429299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4953.522264] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 4956.740693] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4958.127906] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4960.246591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4960.250643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4960.276445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4961.251842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4962.578841] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4962.652157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4965.344812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4965.346530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.347847] LustreError: Skipped 1 previous similar message [ 4965.348276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4965.352609] Lustre: Skipped 1 previous similar message [ 4969.356797] Lustre: server umount lustre-OST0000 complete [ 4974.047887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4974.050845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.055480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4974.057609] Lustre: Skipped 1 previous similar message [ 4975.584175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4975.584683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.665725] Lustre: server umount lustre-MDT0000 complete [ 4977.995170] LustreError: 302879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778283032 with bad export cookie 7491000763093273133 [ 4977.997376] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4978.000522] LustreError: 302879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4978.129133] Lustre: server umount lustre-MDT0001 complete [ 4981.354384] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:30:35 (1778283035) [ 4986.611579] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 4989.829864] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5001.018868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.102971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5001.113290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5001.149033] Lustre: lustre-MDT0000: new disk, initializing [ 5001.170991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5001.177713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5002.383787] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5006.544042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5006.582808] Lustre: 308428:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5006.603349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5006.605679] Lustre: Skipped 1 previous similar message [ 5006.639790] Lustre: lustre-MDT0001: new disk, initializing [ 5006.665926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5006.675963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5006.678919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5007.910720] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5010.988476] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5012.240834] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5014.575612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5014.671497] Lustre: lustre-OST0000: new disk, initializing [ 5014.673485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5014.699897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5016.496713] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5016.695215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5016.698221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5016.724480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5019.528195] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5032.418152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5032.418322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.420381] LustreError: Skipped 1 previous similar message [ 5032.426757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5032.430420] Lustre: Skipped 2 previous similar messages [ 5034.575304] Lustre: server umount lustre-OST0000 complete [ 5035.747441] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5037.535994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5037.536351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.538046] Lustre: Skipped 1 previous similar message [ 5037.541179] Lustre: Skipped 2 previous similar messages [ 5047.776285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5047.778252] Lustre: Skipped 3 previous similar messages [ 5050.335095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5050.390926] Lustre: server umount lustre-MDT0000 complete [ 5051.603130] LustreError: 310169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778283106 with bad export cookie 7491000763093274001 [ 5051.606095] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5051.606632] LustreError: 310169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5051.721705] Lustre: server umount lustre-MDT0001 complete [ 5055.347335] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5056.295041] Key type lgssc unregistered [ 5056.411365] LNet: 312194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5056.413658] LNetError: 312194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5056.421246] LNet: Removed LNI 192.168.203.123@tcp [ 5056.690490] Key type .llcrypt unregistered [ 5056.691665] Key type ._llcrypt unregistered [ 5087.018567] Key type ._llcrypt registered [ 5087.019553] Key type .llcrypt registered [ 5087.051671] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5087.143700] Key type .llcrypt unregistered [ 5087.145273] Key type ._llcrypt unregistered [ 5113.234416] Key type ._llcrypt registered [ 5113.235258] Key type .llcrypt registered [ 5113.278502] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5113.382525] Key type .llcrypt unregistered [ 5113.383518] Key type ._llcrypt unregistered [ 5124.433520] Key type ._llcrypt registered [ 5124.434711] Key type .llcrypt registered [ 5124.468555] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5124.567362] Key type .llcrypt unregistered [ 5124.568743] Key type ._llcrypt unregistered [ 5134.799435] Key type ._llcrypt registered [ 5134.800514] Key type .llcrypt registered [ 5134.839696] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5134.939321] Key type .llcrypt unregistered [ 5134.940521] Key type ._llcrypt unregistered [ 5161.205131] Key type ._llcrypt registered [ 5161.207066] Key type .llcrypt registered [ 5161.250340] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5161.365345] Key type .llcrypt unregistered [ 5161.366452] Key type ._llcrypt unregistered [ 5172.438056] Key type ._llcrypt registered [ 5172.439370] Key type .llcrypt registered [ 5172.480573] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5172.579175] Key type .llcrypt unregistered [ 5172.580491] Key type ._llcrypt unregistered [ 5183.587796] Key type ._llcrypt registered [ 5183.589631] Key type .llcrypt registered [ 5183.633333] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5183.732550] Key type .llcrypt unregistered [ 5183.734117] Key type ._llcrypt unregistered [ 5194.864818] Key type ._llcrypt registered [ 5194.866031] Key type .llcrypt registered [ 5194.915230] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5195.025278] Key type .llcrypt unregistered [ 5195.026309] Key type ._llcrypt unregistered [ 5206.107464] Key type ._llcrypt registered [ 5206.108563] Key type .llcrypt registered [ 5206.143267] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5206.238392] Key type .llcrypt unregistered [ 5206.239510] Key type ._llcrypt unregistered [ 5216.936157] Key type ._llcrypt registered [ 5216.937696] Key type .llcrypt registered [ 5216.977980] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5217.082424] Key type .llcrypt unregistered [ 5217.084290] Key type ._llcrypt unregistered [ 5238.029643] Key type ._llcrypt registered [ 5238.030749] Key type .llcrypt registered [ 5238.066842] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5238.167431] Key type .llcrypt unregistered [ 5238.169438] Key type ._llcrypt unregistered [ 5249.218717] Key type ._llcrypt registered [ 5249.220188] Key type .llcrypt registered [ 5249.263644] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5249.373313] Key type .llcrypt unregistered [ 5249.374435] Key type ._llcrypt unregistered [ 5260.188860] Key type ._llcrypt registered [ 5260.190612] Key type .llcrypt registered [ 5260.233501] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5260.340210] Key type .llcrypt unregistered [ 5260.341444] Key type ._llcrypt unregistered [ 5277.936520] Key type ._llcrypt registered [ 5277.937564] Key type .llcrypt registered [ 5277.980368] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5284.056924] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5284.339296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5284.347224] alg: No test for adler32 (adler32-zlib) [ 5285.199095] Lustre: Lustre: Build Version: 2.17.52_6_gcfb9284 [ 5285.289335] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 5286.879112] Key type lgssc registered [ 5287.252953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5298.396828] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5298.404194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5299.487839] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5299.498179] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5299.537684] Lustre: tst250fs-MDT0000: new disk, initializing [ 5299.562519] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5299.568952] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5300.872033] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5304.993973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.036392] Lustre: 331816:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5305.059967] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5305.062097] Lustre: Skipped 1 previous similar message [ 5305.073231] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (not set up) [ 5305.099352] Lustre: tst250fs-MDT0001: new disk, initializing [ 5305.127335] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5305.139122] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5305.142392] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5306.394365] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5308.685937] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5310.994076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5311.094672] Lustre: tst250fs-OST0000: new disk, initializing [ 5311.097337] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5311.129446] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.136969] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5317.278773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5317.343078] Lustre: tst250fs-OST0001: new disk, initializing [ 5317.345272] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5317.378086] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5318.534928] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5318.539222] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5318.606074] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5318.613120] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5319.373323] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug -1 all [ 5321.981397] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5333.990180] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5333.991855] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5333.994680] Lustre: Skipped 2 previous similar messages [ 5333.999354] Lustre: Skipped 3 previous similar messages [ 5338.265444] Lustre: server umount tst250fs-MDT0000 complete [ 5339.510159] LustreError: 331810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778283394 with bad export cookie 6091792027657347998 [ 5339.512705] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5339.515042] LustreError: 331810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5339.635533] Lustre: server umount tst250fs-MDT0001 complete [ 5350.404473] Lustre: server umount tst250fs-OST0000 complete [ 5361.891312] Lustre: server umount tst250fs-OST0001 complete [ 5368.170752] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5371.069685] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5381.312082] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 19:37:15 (1778283435) [ 5381.843120] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5386.768229] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 5389.514829] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 5399.524846] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5093 sec ========== 19:37:33 (1778283453) [ 5400.191666] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:37:34 (1778283454) === [ 5401.693676] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:37:35 (1778283455) === [ 5422.666477] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [ 5423.932592] Key type lgssc unregistered [ 5424.097714] LNet: 343273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5424.103630] LNetError: 343273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5424.114776] LNet: Removed LNI 192.168.203.123@tcp [ 5424.503282] Key type .llcrypt unregistered [ 5424.505029] Key type ._llcrypt unregistered