[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468557301 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2576MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003048] x2apic enabled [ 0.004000] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.005589] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007014] pid_max: default: 32768 minimum: 301 [ 0.009029] LSM: Security Framework initializing [ 0.011047] Yama: becoming mindful. [ 0.012067] SELinux: Initializing. [ 0.014041] *** VALIDATE selinux *** [ 0.026000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.035176] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.037030] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.038120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.039000] *** VALIDATE tmpfs *** [ 0.040624] *** VALIDATE proc *** [ 0.042082] *** VALIDATE cgroup *** [ 0.043014] *** VALIDATE cgroup2 *** [ 0.045298] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.046176] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.047014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.048037] Spectre V2 : User space: Vulnerable [ 0.049015] Speculative Store Bypass: Vulnerable [ 0.052154] debug: unmapping init [mem 0xffffffffb8859000-0xffffffffb8860fff] [ 0.056473] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.057659] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.058105] ... version: 2 [ 0.059018] ... bit width: 48 [ 0.060019] ... generic registers: 4 [ 0.061017] ... value mask: 0000ffffffffffff [ 0.063000] ... max period: 00007fffffffffff [ 0.063020] ... fixed-purpose events: 3 [ 0.064017] ... event mask: 000000070000000f [ 0.065326] rcu: Hierarchical SRCU implementation. [ 0.069615] smp: Bringing up secondary CPUs ... [ 0.072065] x86: Booting SMP configuration: [ 0.073020] .... node #0, CPUs: #1 #2 #3 [ 0.097127] smp: Brought up 1 node, 4 CPUs [ 0.099037] smpboot: Max logical packages: 1 [ 0.100019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149034] node 0 deferred pages initialised in 42ms [ 0.166142] devtmpfs: initialized [ 0.170295] x86/mm: Memory block size: 128MB [ 0.183310] gcov: version magic: 0x41383552 [ 0.186643] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.195240] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.199298] pinctrl core: initialized pinctrl subsystem [ 0.200000] [ 0.202010] ************************************************************* [ 0.205011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.214015] ** ** [ 0.222024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.229053] ** ** [ 0.237058] ** This means that this kernel is built to expose internal ** [ 0.239023] ** IOMMU data structures, which may compromise security on ** [ 0.246016] ** your system. ** [ 0.256016] ** ** [ 0.261060] ** If you see this message and you are not debugging the ** [ 0.266015] ** kernel, report this immediately to your vendor! ** [ 0.271021] ** ** [ 0.274023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.279015] ************************************************************* [ 0.285082] NET: Registered protocol family 16 [ 0.289268] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.293085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.299134] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.304107] cpuidle: using governor menu [ 0.306561] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.311018] PCI: Using configuration type 1 for base access [ 0.314295] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.327193] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.328021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.331102] cryptd: max_cpu_qlen set to 1000 [ 0.338262] ACPI: Added _OSI(Module Device) [ 0.340015] ACPI: Added _OSI(Processor Device) [ 0.343031] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.346022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.359210] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.381613] ACPI: Interpreter enabled [ 0.384090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.386024] ACPI: Using IOAPIC for interrupt routing [ 0.395111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.408000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.439201] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.441050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.446022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.457091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.482219] acpiphp: Slot [2] registered [ 0.487158] acpiphp: Slot [5] registered [ 0.488000] acpiphp: Slot [6] registered [ 0.491432] acpiphp: Slot [7] registered [ 0.496152] acpiphp: Slot [8] registered [ 0.505215] acpiphp: Slot [9] registered [ 0.506000] acpiphp: Slot [10] registered [ 0.510238] acpiphp: Slot [3] registered [ 0.512939] acpiphp: Slot [4] registered [ 0.521135] acpiphp: Slot [11] registered [ 0.522000] acpiphp: Slot [12] registered [ 0.528222] acpiphp: Slot [13] registered [ 0.534108] acpiphp: Slot [14] registered [ 0.542104] acpiphp: Slot [15] registered [ 0.543111] acpiphp: Slot [16] registered [ 0.544091] acpiphp: Slot [17] registered [ 0.551643] acpiphp: Slot [18] registered [ 0.552254] acpiphp: Slot [19] registered [ 0.558041] acpiphp: Slot [20] registered [ 0.566045] acpiphp: Slot [21] registered [ 0.571084] acpiphp: Slot [22] registered [ 0.577143] acpiphp: Slot [23] registered [ 0.579125] acpiphp: Slot [24] registered [ 0.585000] acpiphp: Slot [25] registered [ 0.588135] acpiphp: Slot [26] registered [ 0.595229] acpiphp: Slot [27] registered [ 0.601000] acpiphp: Slot [28] registered [ 0.604312] acpiphp: Slot [29] registered [ 0.608126] acpiphp: Slot [30] registered [ 0.612123] acpiphp: Slot [31] registered [ 0.618102] PCI host bridge to bus 0000:00 [ 0.620025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.630025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.637026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.647031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.659026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.673028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.682319] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.690000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.696408] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.731613] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.749058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.754017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.758019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.762019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.768866] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.778301] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.781426] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.786733] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.797079] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.815016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.818014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.824537] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.832019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.854019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.926016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.955000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.976020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.001018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.072018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 1.091000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.123035] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.134017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.158017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.187572] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.207019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.220020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.265016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.288374] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.301016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.306015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.345017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.377672] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.398015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.411017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.457018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.486634] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.487380] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.492776] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.499043] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.503922] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.511136] iommu: Default domain type: Passthrough [ 1.514603] SCSI subsystem initialized [ 1.521248] ACPI: bus type USB registered [ 1.523135] usbcore: registered new interface driver usbfs [ 1.529103] usbcore: registered new interface driver hub [ 1.530084] usbcore: registered new device driver usb [ 1.535582] pps_core: LinuxPPS API ver. 1 registered [ 1.540014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.547110] PTP clock support registered [ 1.552264] EDAC MC: Ver: 3.0.0 [ 1.553141] PCI: Using ACPI for IRQ routing [ 1.557510] NetLabel: Initializing [ 1.560050] NetLabel: domain hash size = 128 [ 1.563011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.568095] NetLabel: unlabeled traffic allowed by default [ 1.573473] vgaarb: loaded [ 1.584085] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.585011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.599618] clocksource: Switched to clocksource kvm-clock [ 1.845237] VFS: Disk quotas dquot_6.6.0 [ 1.848741] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.852131] *** VALIDATE ramfs *** [ 1.860982] *** VALIDATE hugetlbfs *** [ 1.865724] pnp: PnP ACPI init [ 1.869511] pnp: PnP ACPI: found 6 devices [ 1.934708] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.943879] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.950981] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.957394] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.962157] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.966712] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.973482] NET: Registered protocol family 2 [ 1.976257] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.996774] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 2.006674] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 2.037334] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 2.042763] TCP: Hash tables configured (established 65536 bind 65536) [ 2.050867] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 2.059340] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.063903] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.068206] NET: Registered protocol family 1 [ 2.086245] RPC: Registered named UNIX socket transport module. [ 2.087756] RPC: Registered udp transport module. [ 2.089044] RPC: Registered tcp transport module. [ 2.090804] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.092884] NET: Registered protocol family 44 [ 2.094826] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.096810] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.099283] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.102464] PCI: CLS 0 bytes, default 64 [ 2.104491] Unpacking initramfs... [ 5.604797] debug: unmapping init [mem 0xffff8f047cc54000-0xffff8f047ffbffff] [ 5.624738] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.634244] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 5.649156] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 8.861119] Initialise system trusted keyrings [ 8.872784] Key type blacklist registered [ 8.879352] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 8.907462] zbud: loaded [ 8.919428] *** VALIDATE nfs *** [ 8.924333] *** VALIDATE nfs4 *** [ 8.932454] pstore: using deflate compression [ 8.940355] Platform Keyring initialized [ 9.281175] NET: Registered protocol family 38 [ 9.287565] Key type asymmetric registered [ 9.288985] Asymmetric key parser 'x509' registered [ 9.294300] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 9.307265] io scheduler mq-deadline registered [ 9.315590] io scheduler kyber registered [ 9.317126] io scheduler bfq registered [ 9.327080] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 9.337577] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 9.349768] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 9.356643] ACPI: Power Button [PWRF] [ 9.374458] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 9.390566] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 9.436338] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 9.455157] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 9.503635] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 9.566100] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.608948] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.629052] Non-volatile memory driver v1.3 [ 9.634245] Linux agpgart interface v0.103 [ 9.730811] virtio_blk virtio1: [vda] 134360 512-byte logical blocks (68.8 MB/65.6 MiB) [ 9.747182] vda: detected capacity change from 0 to 68792320 [ 9.843597] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.860409] vdb: detected capacity change from 0 to 1073741824 [ 9.911513] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.924588] vdc: detected capacity change from 0 to 2621440000 [ 9.943244] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.946353] vdd: detected capacity change from 0 to 2621440000 [ 9.969145] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.976062] vde: detected capacity change from 0 to 4294967296 [ 9.998650] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 10.002592] vdf: detected capacity change from 0 to 4294967296 [ 10.017422] libphy: Fixed MDIO Bus: probed [ 10.036408] usbcore: registered new interface driver usbserial_generic [ 10.052104] usbserial: USB Serial support registered for generic [ 10.058227] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 10.065519] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 10.068486] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 10.074926] mousedev: PS/2 mouse device common for all mice [ 10.083620] rtc_cmos 00:05: RTC can wake from S4 [ 10.088303] rtc_cmos 00:05: registered as rtc0 [ 10.089764] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 10.093590] intel_pstate: CPU model not supported [ 10.097598] hid: raw HID events driver (C) Jiri Kosina [ 10.100167] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 10.105258] usbcore: registered new interface driver usbhid [ 10.105266] usbhid: USB HID core driver [ 10.105413] drop_monitor: Initializing network drop monitor service [ 10.105918] Initializing XFRM netlink socket [ 10.106507] NET: Registered protocol family 10 [ 10.124515] Segment Routing with IPv6 [ 10.138763] NET: Registered protocol family 17 [ 10.141165] mpls_gso: MPLS GSO support [ 10.148473] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 10.153409] RAS: Correctable Errors collector initialized. [ 10.157516] AVX version of gcm_enc/dec engaged. [ 10.158762] AES CTR mode by8 optimization enabled [ 10.161502] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 10.347383] sched_clock: Marking stable (10347330562, 0)->(11857446606, -1510116044) [ 10.352779] registered taskstats version 1 [ 10.364364] Loading compiled-in X.509 certificates [ 10.371249] zswap: loaded using pool lzo/zbud [ 10.444272] Key type big_key registered [ 10.475698] Key type encrypted registered [ 10.476945] ima: No TPM chip found, activating TPM-bypass! [ 10.480738] ima: Allocated hash algorithm: sha1 [ 10.487402] ima: No architecture policies found [ 10.488765] evm: Initialising EVM extended attributes: [ 10.490025] evm: security.selinux [ 10.490866] evm: security.ima [ 10.493578] evm: security.capability [ 10.494443] evm: HMAC attrs: 0x1 [ 10.497978] rtc_cmos 00:05: setting system clock to 2026-01-30 06:24:30 UTC (1769754270) [ 10.524251] debug: unmapping init [mem 0xffffffffb9803000-0xffffffffb99fffff] [ 10.532701] debug: unmapping init [mem 0xffffffffb8582000-0xffffffffb8858fff] [ 10.543455] Write protecting the kernel read-only data: 28672k [ 10.548627] debug: unmapping init [mem 0xffffffffb6c03000-0xffffffffb6dfffff] [ 10.553654] debug: unmapping init [mem 0xffffffffb7514000-0xffffffffb75fffff] [ 10.668341] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.682259] systemd[1]: Detected virtualization kvm. [ 10.687415] systemd[1]: Detected architecture x86-64. [ 10.690905] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.771809] systemd[1]: No hostname configured. [ 10.775347] systemd[1]: Set hostname to . [ 10.780429] random: systemd: uninitialized urandom read (16 bytes read) [ 10.787634] systemd[1]: Initializing machine ID from random generator. [ 10.960708] random: ln: uninitialized urandom read (6 bytes read) [ 11.127108] random: systemd: uninitialized urandom read (16 bytes read) [ 11.130574] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 11.148935] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 11.158080] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 13.032412] device-mapper: uevent: version 1.0.3 [ 13.041455] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 15.575835] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 15.792261] scsi host0: ata_piix [ 15.795360] random: fast init done [ 15.864086] scsi host1: ata_piix [ 15.866691] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 15.869126] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 22.011862] random: crng init done [ 22.022074] random: 7 urandom warning(s) missed due to ratelimiting [ 23.359580] dracut-initqueue[597]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 24.227192] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.718443] printk: systemd: 26 output lines suppressed due to ratelimiting [ 27.467306] SELinux: Disabled at runtime. [ 27.572762] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.580882] systemd[1]: Detected virtualization kvm. [ 27.581771] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.623655] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.628677] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.639887] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.647387] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.652163] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.660689] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.676966] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ 29.139216] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 30.167237] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.082660] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.149454] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.656895] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.740925] EDAC sbridge: Ver: 1.1.2 [ 34.979346] Key type dns_resolver registered [ 35.609232] NFS: Registering the id_resolver key type [ 35.611613] Key type id_resolver registered [ 35.614090] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg646-server login: [ 94.009061] hrtimer: interrupt took 11915970 ns [ 94.511970] libcfs: loading out-of-tree module taints kernel. [ 94.575166] Key type ._llcrypt registered [ 94.577160] Key type .llcrypt registered [ 94.704347] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 115.221846] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 117.572658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.582669] alg: No test for adler32 (adler32-zlib) [ 118.834310] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 119.738217] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 121.464218] Key type lgssc registered [ 123.702812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.300095] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 188.367533] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 202.054343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 202.087097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.307330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 203.347595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 203.544160] Lustre: lustre-MDT0000: new disk, initializing [ 203.703492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.751561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 208.582623] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.801406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.924378] Lustre: 6516:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.948965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 222.952109] Lustre: Skipped 1 previous similar message [ 223.036200] Lustre: lustre-MDT0001: new disk, initializing [ 223.097646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.125051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 223.138824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 228.058124] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.901498] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 243.285158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 243.624122] Lustre: lustre-OST0000: new disk, initializing [ 243.635636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 243.740457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.449096] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.449828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 252.456716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 252.555113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 266.058368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 266.208137] Lustre: lustre-OST0001: new disk, initializing [ 266.218229] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 266.308385] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.504665] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 272.924557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 272.937266] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 273.011506] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 284.261838] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 292.426040] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 304.044542] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing check_logdir /tmp/testlogs/ [ 309.725236] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing yml_node [ 313.693035] Lustre: DEBUG MARKER: Client: 2.17.50.86 [ 316.388983] Lustre: DEBUG MARKER: MDS: 2.17.50.86 [ 319.154615] Lustre: DEBUG MARKER: OSS: 2.17.50.86 [ 320.936434] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 30 01:29:39 EST 2026 [ 339.335634] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 341.230452] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.905595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.917425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.934804] Lustre: Skipped 1 previous similar message [ 363.123712] LustreError: 12212:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 363.399236] Lustre: server umount lustre-MDT0000 complete [ 366.975910] LustreError: 6508:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754626 with bad export cookie 4810075237898949082 [ 366.977286] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.981587] LustreError: 6508:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 370.144836] LustreError: 10446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.158054] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.164073] LustreError: 10446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 370.197165] Lustre: Skipped 2 previous similar messages [ 370.210046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.215392] Lustre: Skipped 4 previous similar messages [ 371.170725] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.191147] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 373.189425] LustreError: 12414:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 373.199462] LustreError: 12414:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 373.461935] Lustre: server umount lustre-MDT0001 complete [ 377.286228] LustreError: 12615:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 377.288922] LustreError: 12615:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 377.433310] Lustre: server umount lustre-OST0000 complete [ 381.570118] LustreError: 12816:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 381.573345] LustreError: 12816:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 381.652420] Lustre: server umount lustre-OST0001 complete [ 389.658889] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 400.182888] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 441.482766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.792514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 441.834415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 441.936409] Lustre: lustre-MDT0000: new disk, initializing [ 442.056776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.069883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 446.838177] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 458.159125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.316185] Lustre: 15003:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 458.371563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 458.381063] Lustre: Skipped 1 previous similar message [ 458.479247] Lustre: lustre-MDT0001: new disk, initializing [ 458.653115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 458.758798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 458.771811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 463.592884] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 471.526781] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 475.733387] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 483.126576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.387434] Lustre: lustre-OST0000: new disk, initializing [ 483.391201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 483.502631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.179026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 485.200411] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 485.332118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 489.650128] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 497.396962] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 501.763550] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 502.065821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 506.983918] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 507.253527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 510.955534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.970444] Lustre: Skipped 1 previous similar message [ 510.972472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.978537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.978546] Lustre: Skipped 2 previous similar messages [ 511.001479] LustreError: Skipped 1 previous similar message [ 515.612978] LustreError: 17937:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 515.618103] LustreError: 17937:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 515.758463] Lustre: server umount lustre-OST0000 complete [ 521.186751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.210783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 521.220056] Lustre: Skipped 2 previous similar messages [ 521.228664] Lustre: Skipped 1 previous similar message [ 526.309358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 526.322688] Lustre: Skipped 2 previous similar messages [ 526.471984] LustreError: 18139:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 526.478976] LustreError: 18139:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 526.684471] Lustre: server umount lustre-MDT0000 complete [ 530.693984] LustreError: 15780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754790 with bad export cookie 4810075237898950608 [ 530.694459] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.707918] LustreError: 15780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 531.061534] Lustre: server umount lustre-MDT0001 complete [ 538.396307] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:33:15 (1769754795) [ 563.430239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.196305] LustreError: 19478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.356681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 569.825473] LustreError: 19478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.343746] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 574.946588] LustreError: 19478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.168825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.077949] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 591.389479] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 595.615429] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 598.720764] Lustre: Failing over lustre-MDT0000 [ 598.845199] LustreError: 20819:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 598.849193] LustreError: 20819:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 598.974027] Lustre: server umount lustre-MDT0000 complete [ 601.071149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 601.078468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.097148] LustreError: 19478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.097159] LustreError: 19478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 601.185734] Lustre: Skipped 1 previous similar message [ 617.952157] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769754861/real 1769754861] req@ffff8f03c30b7b80 x1855721967873280/t0(0) o400->MGC192.168.206.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769754877 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 617.989610] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.007447] LustreError: 19478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.035300] LustreError: 19478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 619.585854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.193234] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8f03c30b4380 x1855721967876096/t0(0) o250->MGC192.168.206.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 628.553668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.557187] Lustre: Skipped 1 previous similar message [ 628.623251] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 633.743336] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 633.829780] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 633.845941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 633.910621] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 642.244666] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 646.298578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.304042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 646.315670] Lustre: Skipped 1 previous similar message [ 646.337893] Lustre: Skipped 1 previous similar message [ 652.547897] LustreError: 21801:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 652.559891] LustreError: 21801:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 652.745776] Lustre: server umount lustre-MDT0000 complete [ 655.842334] LustreError: 19477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.864426] LustreError: 19477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 656.790800] LustreError: 19459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769754916 with bad export cookie 4810075237898952029 [ 656.799788] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.803972] LustreError: 19459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 657.298274] Lustre: server umount lustre-MDT0001 complete [ 666.004577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 671.684388] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 682.049715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.852798] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 694.346858] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 698.619445] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 701.389127] Lustre: Failing over lustre-MDT0000 [ 701.703490] Lustre: server umount lustre-MDT0000 complete [ 702.947687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.974743] Lustre: Skipped 1 previous similar message [ 718.309129] Lustre: 3668:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769754962/real 1769754962] req@ffff8f04ff77b800 x1855721967903104/t0(0) o400->MGC192.168.206.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769754978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 718.356220] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.947455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.426099] LustreError: 23192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 723.448189] LustreError: 23192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 729.242565] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 733.588757] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 734.181528] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 734.204844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 734.210917] Lustre: Skipped 1 previous similar message [ 734.225206] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 742.051181] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 748.517058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.537449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 748.549878] Lustre: Skipped 2 previous similar messages [ 752.420935] LustreError: 24800:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 752.424515] LustreError: 24800:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 752.564631] Lustre: server umount lustre-MDT0000 complete [ 757.084787] LustreError: 22460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755017 with bad export cookie 4810075237898952967 [ 757.098824] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 757.101970] LustreError: 22460:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 757.532152] Lustre: server umount lustre-MDT0001 complete [ 768.691250] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:37:06 (1769755026) [ 787.250830] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 799.091692] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 845.937324] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 858.497863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.843827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 858.892704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 859.001668] Lustre: lustre-MDT0000: new disk, initializing [ 859.098704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.112612] Lustre: Skipped 3 previous similar messages [ 859.128030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 863.611376] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 874.892658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 874.978351] Lustre: 28463:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 875.395789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 881.076681] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 889.762366] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 894.172727] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 901.483868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 903.003640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 903.043430] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 903.154517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 903.160133] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 908.196447] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 916.177149] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 922.648470] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 949.218132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 949.233287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.240397] Lustre: Skipped 1 previous similar message [ 949.248320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 949.253339] Lustre: Skipped 1 previous similar message [ 950.965082] LustreError: 31117:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 950.967659] LustreError: 31117:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 951.042729] Lustre: server umount lustre-OST0000 complete [ 957.409685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 957.411861] LustreError: Skipped 1 previous similar message [ 961.827947] Lustre: server umount lustre-MDT0000 complete [ 964.578726] LustreError: 30212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.601629] LustreError: 30212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 965.628203] LustreError: 28454:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755225 with bad export cookie 4810075237898953373 [ 965.633200] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 965.635953] LustreError: 28454:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 966.065539] Lustre: server umount lustre-MDT0001 complete [ 980.193534] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 984.094397] Key type lgssc unregistered [ 984.395918] LNet: 32056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.408327] LNetError: 32056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.420659] LNet: Removed LNI 192.168.206.146@tcp [ 985.615518] Key type .llcrypt unregistered [ 985.618685] Key type ._llcrypt unregistered [ 1014.091126] Key type ._llcrypt registered [ 1014.093819] Key type .llcrypt registered [ 1014.204664] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1015.350901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.602332] alg: No test for adler32 (adler32-zlib) [ 1017.088711] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 1017.399092] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1019.272369] Key type lgssc registered [ 1020.881496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1030.712682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1030.747481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.614536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.395993] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1042.921253] LustreError: 33450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.962668] LustreError: 33450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1048.041587] LustreError: 33450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.804228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1049.299704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1055.099794] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1061.716545] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1066.627695] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1076.906075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1077.149120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.566912] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1089.445170] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1094.593827] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1106.807076] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:42:45 (1769755365) [ 1108.856668] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1110.785315] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1118.496836] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:42:56 (1769755376) [ 1128.379047] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:43:06 (1769755386) [ 1129.637673] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1139.334351] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1139.341777] Lustre: Skipped 2 previous similar messages [ 1141.959458] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:43:19 (1769755399) [ 1143.826783] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1143.831817] Lustre: Skipped 2 previous similar messages [ 1153.001421] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1153.022951] Lustre: Skipped 1 previous similar message [ 1169.984263] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1169.989271] Lustre: Skipped 8 previous similar messages [ 1172.616803] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:43:50 (1769755430) [ 1176.359885] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1181.547439] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1191.485756] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:44:09 (1769755449) [ 1208.286183] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1208.292353] Lustre: Skipped 4 previous similar messages [ 1210.583403] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:44:28 (1769755468) [ 1231.841638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1231.858698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.886944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.867411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1232.874484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.876548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.896033] Lustre: Skipped 1 previous similar message [ 1234.576251] LustreError: 40030:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1234.730394] Lustre: server umount lustre-OST0000 complete [ 1239.014684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1239.028658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.052688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1243.109119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.110685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1243.135950] Lustre: Skipped 1 previous similar message [ 1248.228098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1248.247927] Lustre: Skipped 1 previous similar message [ 1253.344261] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1253.505633] LustreError: 40231:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1253.510783] LustreError: 40231:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1253.916844] Lustre: server umount lustre-MDT0000 complete [ 1257.442097] LustreError: 33447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.472317] LustreError: 33447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1259.941347] LustreError: 33430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755519 with bad export cookie 14717699944056879477 [ 1259.944174] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1259.947738] LustreError: 33430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1260.161801] LustreError: 40434:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1260.171985] LustreError: 40434:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1260.863515] Lustre: server umount lustre-MDT0001 complete [ 1275.285294] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1278.918851] Key type lgssc unregistered [ 1279.324510] LNet: 40967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.333148] LNetError: 40967:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.361295] LNet: Removed LNI 192.168.206.146@tcp [ 1280.456311] Key type .llcrypt unregistered [ 1280.459684] Key type ._llcrypt unregistered [ 1302.279467] Key type ._llcrypt registered [ 1302.280989] Key type .llcrypt registered [ 1302.464637] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 1323.969122] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1325.995943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1326.023182] alg: No test for adler32 (adler32-zlib) [ 1327.172284] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 1327.487483] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1329.280555] Key type lgssc registered [ 1330.685522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1371.166585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1371.208746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.702871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1372.761532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1372.952372] Lustre: lustre-MDT0000: new disk, initializing [ 1373.192607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.221948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1379.467984] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1391.387927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.533540] Lustre: 44090:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1391.585164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1391.590294] Lustre: Skipped 1 previous similar message [ 1391.664440] Lustre: lustre-MDT0001: new disk, initializing [ 1391.836637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1391.881515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1391.896510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1397.301902] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1405.086375] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1409.323539] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1416.950632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1417.326492] Lustre: lustre-OST0000: new disk, initializing [ 1417.330189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1417.436780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1424.429265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1424.454516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1424.557739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1424.712713] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1433.423719] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1438.427672] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1438.707889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1443.899930] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1444.146315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1450.483760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.493770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1450.512725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1450.525033] Lustre: Skipped 1 previous similar message [ 1452.495132] LustreError: 47022:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1452.775384] Lustre: server umount lustre-OST0000 complete [ 1458.659467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1458.666535] LustreError: Skipped 1 previous similar message [ 1458.670981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.682082] Lustre: Skipped 1 previous similar message [ 1458.686044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.710568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.714645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.762072] Lustre: Skipped 1 previous similar message [ 1463.098161] LustreError: 47224:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1463.101894] LustreError: 47224:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1463.229444] Lustre: server umount lustre-MDT0000 complete [ 1467.172247] LustreError: 44082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755727 with bad export cookie 9599819373793484259 [ 1467.176568] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1467.182020] LustreError: 44082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1467.354605] LustreError: 47425:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1467.361276] LustreError: 47425:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1467.636335] Lustre: server umount lustre-MDT0001 complete [ 1476.929749] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:48:55 (1769755735) [ 1484.876439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1485.516656] LustreError: 48145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.540654] LustreError: 48145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1485.601497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.944221] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1490.913434] LustreError: 48145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1495.037112] LustreError: 48145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.330839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.782579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1503.746421] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1508.972600] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1512.506959] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1519.685127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1520.039411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.829810] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1532.179797] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1536.769619] LustreError: 50337:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1537.563523] Lustre: Modifying parameter general.timeout=1 in log params [ 1539.163516] Lustre: Modifying parameter general.timeout=3 in log params [ 1539.171193] Lustre: Skipped 1 previous similar message [ 1541.482427] Lustre: Modifying parameter general.timeout=6 in log params [ 1541.486032] Lustre: Skipped 2 previous similar messages [ 1545.561753] Lustre: Modifying parameter general.timeout=11 in log params [ 1545.565233] Lustre: Skipped 4 previous similar messages [ 1554.245963] Lustre: Modifying parameter general.timeout=21 in log params [ 1554.252973] Lustre: Skipped 9 previous similar messages [ 1570.313110] Lustre: Modifying parameter general.timeout=40 in log params [ 1570.315442] Lustre: Skipped 18 previous similar messages [ 1602.479948] Lustre: Modifying parameter general.timeout=76 in log params [ 1602.485560] Lustre: Skipped 35 previous similar messages [ 1663.676209] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:52:02 (1769755922) [ 1666.324570] Lustre: 57280:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1667.308419] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1667.315482] Lustre: Skipped 56 previous similar messages [ 1668.202432] Lustre: 57378:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1671.760514] Lustre: 57592:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1671.767588] Lustre: 57592:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1700.728548] Lustre: 59161:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1700.737288] Lustre: 59161:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 1700.756704] LustreError: 59161:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1704.377907] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:52:42 (1769755962) [ 1717.737330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1717.748046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1717.764873] Lustre: Skipped 2 previous similar messages [ 1717.770077] Lustre: Skipped 2 previous similar messages [ 1722.221404] LustreError: 59862:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1722.246313] LustreError: 59862:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1722.412237] Lustre: server umount lustre-MDT0000 complete [ 1722.852777] LustreError: 48144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.868152] LustreError: 48144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1725.945449] LustreError: 48127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755985 with bad export cookie 9599819373793485141 [ 1725.952602] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1725.957898] LustreError: 48127:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1726.278117] Lustre: server umount lustre-MDT0001 complete [ 1739.622367] LustreError: 60264:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1739.624971] LustreError: 60264:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1739.976713] Lustre: server umount lustre-OST0000 complete [ 1766.554093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.840246] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1767.245844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1770.437919] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1778.963340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.089325] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1779.149333] Lustre: 61692:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1784.592514] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1791.603527] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1796.433220] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1803.484341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1803.674353] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1803.827637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.832833] Lustre: Skipped 1 previous similar message [ 1810.096843] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1816.237902] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1817.733821] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1817.739984] Lustre: Skipped 20 previous similar messages [ 1817.834576] Lustre: 63920:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1826.279374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1826.280534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1826.286459] LustreError: Skipped 1 previous similar message [ 1826.287156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1826.330341] Lustre: Skipped 1 previous similar message [ 1830.364116] LustreError: 64176:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1830.367705] LustreError: 64176:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1830.531027] Lustre: server umount lustre-OST0000 complete [ 1836.007708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1836.027157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1836.064162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1836.066984] Lustre: Skipped 1 previous similar message [ 1841.644370] LustreError: 61704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1841.825502] Lustre: server umount lustre-MDT0000 complete [ 1846.153736] LustreError: 63328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756106 with bad export cookie 9599819373793488200 [ 1846.167148] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1846.176938] LustreError: 63328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1846.488968] LustreError: 64581:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1846.496586] LustreError: 64581:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1846.782991] Lustre: server umount lustre-MDT0001 complete [ 1860.184524] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1864.468790] Key type lgssc unregistered [ 1864.683165] LNet: 65113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.691578] LNetError: 65113:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1865.706390] LNet: Removed LNI 192.168.206.146@tcp [ 1866.899653] Key type .llcrypt unregistered [ 1866.901958] Key type ._llcrypt unregistered [ 1894.389436] Key type ._llcrypt registered [ 1894.391048] Key type .llcrypt registered [ 1894.516646] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1896.184893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1896.266634] alg: No test for adler32 (adler32-zlib) [ 1897.248732] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 1897.452547] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1899.176205] Key type lgssc registered [ 1900.227590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1909.382785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1909.398303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.297357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.337905] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1921.509337] LustreError: 66511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.520317] LustreError: 66511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1924.403519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.118639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1929.248340] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1935.162935] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1938.469670] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1945.273376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1945.890719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.546969] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1957.739655] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1967.358878] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1980.897298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1980.914459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.922528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1981.921541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.923934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1981.938331] Lustre: Skipped 1 previous similar message [ 1981.948882] Lustre: Skipped 2 previous similar messages [ 1983.779220] LustreError: 69218:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1983.903448] Lustre: server umount lustre-MDT0000 complete [ 1987.058345] LustreError: 66508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.091830] LustreError: 66508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1987.445487] LustreError: 66492:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756247 with bad export cookie 2303880511830457999 [ 1987.465202] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1987.624073] LustreError: 69420:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1987.634721] LustreError: 69420:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1987.906480] Lustre: server umount lustre-MDT0001 complete [ 2002.079840] LustreError: 69620:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2002.089812] LustreError: 69620:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2002.232724] Lustre: server umount lustre-OST0000 complete [ 2029.733320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.189256] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2030.217952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2030.656923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.478877] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2044.199957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.298822] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2044.324911] Lustre: 71048:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2044.821129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2048.522927] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2053.957900] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2057.477803] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2064.315070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2064.513869] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2064.918530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.028909] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2076.055145] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2077.432469] Lustre: 73282:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2079.415459] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2087.908872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2087.908963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.919723] LustreError: Skipped 1 previous similar message [ 2087.920693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2087.940238] Lustre: Skipped 1 previous similar message [ 2087.959511] Lustre: Skipped 1 previous similar message [ 2090.273105] LustreError: 73540:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2090.287722] LustreError: 73540:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2090.509775] Lustre: server umount lustre-OST0000 complete [ 2095.587705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2095.593539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.602477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2099.672813] LustreError: 73741:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2099.679919] LustreError: 73741:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2099.760776] Lustre: server umount lustre-MDT0000 complete [ 2103.269886] LustreError: 71059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.292048] LustreError: 71059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2103.405977] LustreError: 71039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756363 with bad export cookie 2303880511830458846 [ 2103.418301] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2103.420048] LustreError: 71039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2103.710178] Lustre: server umount lustre-MDT0001 complete [ 2116.087230] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 2118.885841] Key type lgssc unregistered [ 2119.184923] LNet: 74475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.196463] LNetError: 74475:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.228118] LNet: Removed LNI 192.168.206.146@tcp [ 2120.050806] Key type .llcrypt unregistered [ 2120.052296] Key type ._llcrypt unregistered [ 2143.886506] Key type ._llcrypt registered [ 2143.891074] Key type .llcrypt registered [ 2144.040248] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2145.407440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2145.454468] alg: No test for adler32 (adler32-zlib) [ 2146.467236] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 2146.611987] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 2148.232210] Key type lgssc registered [ 2149.096936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2157.188182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2157.218475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.967059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.280595] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2168.301533] LustreError: 75869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.341729] LustreError: 75869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2173.411591] LustreError: 75869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2176.682864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.622439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2182.469475] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2187.901160] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2191.331774] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2197.568401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2198.200785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.456526] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2211.666364] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2217.830504] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2218.666487] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2220.523564] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2220.531819] Lustre: Skipped 1 previous similar message [ 2223.329786] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2223.335215] Lustre: Skipped 2 previous similar messages [ 2227.859874] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2227.864689] Lustre: Skipped 4 previous similar messages [ 2236.401793] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2236.405885] Lustre: Skipped 9 previous similar messages [ 2253.039062] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2253.044154] Lustre: Skipped 19 previous similar messages [ 2285.669795] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2285.671979] Lustre: Skipped 36 previous similar messages [ 2312.192381] Lustre: 83646:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2314.615153] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 02:02:52 (1769756572) [ 2316.021424] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2317.699246] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 02:02:56 (1769756576) [ 2325.482883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2325.491385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.513165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2326.498664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2326.499321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2326.529598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2326.547860] Lustre: Skipped 1 previous similar message [ 2327.614563] LustreError: 83943:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2327.751957] Lustre: server umount lustre-OST0000 complete [ 2331.618290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.627331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2336.225988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2336.233873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.244729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.331260] LustreError: 84145:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2337.347560] LustreError: 84145:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2337.462318] Lustre: server umount lustre-MDT0000 complete [ 2340.812933] LustreError: 75847:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756600 with bad export cookie 3409365212600204212 [ 2340.819023] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2340.826158] LustreError: 75847:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2340.981356] LustreError: 84346:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2340.988189] LustreError: 84346:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2341.347870] Lustre: server umount lustre-MDT0001 complete [ 2353.034663] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 2356.035497] Key type lgssc unregistered [ 2356.359421] LNet: 84878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.375357] LNetError: 84878:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.399463] LNet: Removed LNI 192.168.206.146@tcp [ 2357.471334] Key type .llcrypt unregistered [ 2357.473707] Key type ._llcrypt unregistered [ 2361.999890] Key type ._llcrypt registered [ 2362.005501] Key type .llcrypt registered [ 2362.114452] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2365.866626] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules [ 2366.887106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.897325] alg: No test for adler32 (adler32-zlib) [ 2367.943865] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 2368.163460] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 2369.816192] Key type lgssc registered [ 2371.090499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2372.439471] LustreError: 86043:0:(lu_object.c:1434:lu_context_key_register()) cfs_race id 60d sleeping [ 2388.181182] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2394.268931] LustreError: 86043:0:(lu_object.c:1434:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2397.284441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2397.306695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.956679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.703125] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2408.420581] LustreError: 87049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.428078] LustreError: 87049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2411.532676] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 02:04:29 (1769756669) [ 2413.538128] LustreError: 87049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.367431] LustreError: 87081:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2416.375554] LustreError: 87081:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2416.380566] LustreError: 87081:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2416.564689] LustreError: 87854:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2417.041890] Lustre: server umount lustre-MDT0000 complete [ 2431.025712] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 2434.056140] Key type lgssc unregistered [ 2434.358067] LNet: 88541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2434.372508] LNetError: 88541:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.431404] LNet: Removed LNI 192.168.206.146@tcp [ 2436.290923] Key type .llcrypt unregistered [ 2436.292097] Key type ._llcrypt unregistered [ 2452.759491] Key type ._llcrypt registered [ 2452.763309] Key type .llcrypt registered [ 2452.859071] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2454.296884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2454.498700] alg: No test for adler32 (adler32-zlib) [ 2455.503413] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 2455.805239] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 2457.552176] Key type lgssc registered [ 2458.792375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2466.615576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2466.658766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.182882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2472.493609] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2478.563628] LustreError: 89741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.585433] LustreError: 89741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2482.136466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.606877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2487.284730] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2492.945832] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2497.552181] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2504.717483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2505.380807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.293761] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2519.429378] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2568.639407] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 02:07:06 (1769756826) [ 2570.702350] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2572.414357] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 02:07:10 (1769756830) [ 2579.943443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2579.948075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.958618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2580.451762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2580.464272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2580.471916] Lustre: Skipped 1 previous similar message [ 2580.488109] Lustre: Skipped 1 previous similar message [ 2583.546980] LustreError: 92655:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2583.677689] Lustre: server umount lustre-MDT0000 complete [ 2585.576982] LustreError: 89735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.600702] LustreError: 89735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2588.386804] LustreError: 89719:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756848 with bad export cookie 10842203835597154868 [ 2588.392169] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2588.403066] LustreError: 89719:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2588.544891] LustreError: 92857:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2588.551875] LustreError: 92857:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2588.767960] Lustre: server umount lustre-MDT0001 complete [ 2602.172038] LustreError: 93058:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2602.190453] LustreError: 93058:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2602.506662] Lustre: server umount lustre-OST0000 complete [ 2614.769053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.418528] LustreError: 93687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.568577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2620.264596] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2620.897621] LustreError: 93687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.014085] LustreError: 93687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.014713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.829413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2634.978251] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2641.783416] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2646.405953] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2660.590826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2660.827912] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2660.838347] LustreError: 93676:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2660.846622] LustreError: 95554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.872720] LustreError: 95554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2660.908843] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2660.921347] LustreError: 95545:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2660.932900] LustreError: 95545:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2660.941941] LustreError: 95545:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2661.048837] Lustre: server umount lustre-OST0000 complete [ 2661.051525] LustreError: 95545:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2670.780733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.967801] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2670.982818] LustreError: 93676:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2671.012469] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2671.019368] LustreError: 96010:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -98 [ 2671.024826] LustreError: 96010:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 2671.029298] LustreError: 96010:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2671.077771] Lustre: server umount lustre-OST0000 complete [ 2671.079627] LustreError: 96010:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2680.278066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2680.535306] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2680.538851] Lustre: Found index 0 for lustre-OST0000, updating log [ 2680.545333] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2680.634868] Lustre: lustre-OST0000: new disk, initializing [ 2680.639253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2680.933744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2686.924191] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2690.654784] Lustre: Failing over lustre-OST0000 [ 2690.704208] LustreError: 97090:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2690.707571] LustreError: 97090:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2690.843107] Lustre: server umount lustre-OST0000 complete [ 2695.653117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.672656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2695.678277] Lustre: Skipped 1 previous similar message [ 2700.772771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2700.779176] Lustre: Skipped 1 previous similar message [ 2704.872865] LustreError: 97290:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2704.888905] LustreError: 97290:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2705.006978] Lustre: server umount lustre-MDT0000 complete [ 2706.920796] LustreError: 94804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.950412] LustreError: 94804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2708.780293] LustreError: 96027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756968 with bad export cookie 10842203835597157619 [ 2708.789437] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2708.792664] LustreError: 96027:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2709.110969] Lustre: server umount lustre-MDT0001 complete [ 2717.318746] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 02:09:35 (1769756975) [ 2732.955629] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2743.278222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.781408] LustreError: 98859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.981561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2748.832342] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2758.311799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2758.625088] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2758.628600] Lustre: Skipped 2 previous similar messages [ 2764.476384] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2767.843969] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2775.268331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.113358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.118958] Lustre: Skipped 1 previous similar message [ 2776.128209] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2783.287078] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2783.295421] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2783.317179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2783.503519] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2784.791119] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 2784.811330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2788.326562] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2788.343156] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2789.375744] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2789.377119] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2789.397314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2793.264650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2793.399276] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2793.488720] Lustre: lustre-OST0001: new disk, initializing [ 2793.492332] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2800.512218] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2802.782398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2802.791637] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2802.925789] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2808.083581] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2815.047212] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2825.724352] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2825.739685] LustreError: Skipped 1 previous similar message [ 2825.741428] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.752465] Lustre: Skipped 1 previous similar message [ 2825.758421] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2830.292220] LustreError: 102094:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2830.302774] LustreError: 102094:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2830.411342] Lustre: server umount lustre-MDT0001 complete [ 2833.895453] LustreError: 99581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.920429] LustreError: 99581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2839.114595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.195369] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2839.200267] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2839.236947] Lustre: 98847:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2840.302208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2840.318092] Lustre: Skipped 1 previous similar message [ 2841.074234] LustreError: 102663:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2841.097794] LustreError: 102663:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2841.107420] LustreError: 98850:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2841.247765] LustreError: 102670:0:(obd_config.c:2075:class_config_llog_handler()) MGC192.168.206.146@tcp: cfg command failed: rc = -17 [ 2841.260122] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2841.260122] [ 2845.177610] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2845.673732] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2855.368268] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 02:11:53 (1769757113) [ 2872.698828] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2891.584600] Lustre: 104697:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2915.198321] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2918.587246] Lustre: 105832:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3004.405278] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.408187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3004.418834] Lustre: Skipped 5 previous similar messages [ 3004.424691] Lustre: Skipped 7 previous similar messages [ 3010.391801] LustreError: 106289:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3010.400579] LustreError: 106289:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3010.504633] Lustre: server umount lustre-MDT0000 complete [ 3014.044595] LustreError: 98838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757274 with bad export cookie 10842203835597158620 [ 3014.046141] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3014.052704] LustreError: 98838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3014.320678] Lustre: server umount lustre-MDT0001 complete [ 3028.734103] Lustre: server umount lustre-OST0000 complete [ 3031.009948] Lustre: 89052:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769757274/real 1769757274] req@ffff8f03caba4a80 x1855724417550080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769757290 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3031.042530] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.129635] Lustre: server umount lustre-OST0001 complete [ 3036.800971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3045.311496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3053.063983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3061.882464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3078.719506] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3088.085291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.101098] Lustre: lustre-MDT0000: reset Object Index mappings [ 3088.539417] LustreError: 109766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.550523] LustreError: 109766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3088.629783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3092.767977] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3101.808681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.836662] Lustre: lustre-MDT0001: reset Object Index mappings [ 3106.742320] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3109.891603] Lustre: 110878:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3117.674352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3125.600374] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3127.685686] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3133.968252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3137.001920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3139.239856] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3139.271759] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3144.948911] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3153.384461] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3157.680201] Lustre: 112730:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3212.053828] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:17:50 (1769757470) [ 3240.422408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.426204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3240.435775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3240.436877] Lustre: Skipped 3 previous similar messages [ 3240.471522] Lustre: Skipped 7 previous similar messages [ 3243.456544] LustreError: 113084:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3243.467468] LustreError: 113084:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3243.578623] Lustre: server umount lustre-MDT0000 complete [ 3247.022612] LustreError: 109746:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757507 with bad export cookie 10842203835597205758 [ 3247.033276] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3247.037651] LustreError: 109746:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3253.656976] Lustre: server umount lustre-MDT0001 complete [ 3263.847240] Lustre: server umount lustre-OST0000 complete [ 3282.400311] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3282.795177] Lustre: server umount lustre-OST0001 complete [ 3289.331671] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 3298.899710] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3337.018373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.273714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3337.316590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3337.401219] Lustre: lustre-MDT0000: new disk, initializing [ 3337.501221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3341.396877] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3347.986924] LustreError: 116438:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3347.999476] LustreError: 116438:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3348.233344] Lustre: server umount lustre-MDT0000 complete [ 3357.594591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.936182] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3357.979940] Lustre: 116954:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3358.002740] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3358.298699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.306115] Lustre: Skipped 4 previous similar messages [ 3362.537751] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3366.910662] Lustre: server umount lustre-MDT0000 complete [ 3376.026645] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:20:34 (1769757634) [ 3378.052971] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3380.041086] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:20:38 (1769757638) [ 3396.139123] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3404.533548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.147619] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3416.004908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.165966] Lustre: 118970:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3416.227150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3416.231467] Lustre: Skipped 1 previous similar message [ 3416.319463] Lustre: lustre-MDT0001: new disk, initializing [ 3416.467831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3416.484462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3421.791409] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3426.661792] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3432.815259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3433.214203] Lustre: lustre-OST0000: new disk, initializing [ 3433.222799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3434.805431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3434.821251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3434.981501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3439.361333] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3449.437394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3449.601937] Lustre: lustre-OST0001: new disk, initializing [ 3449.608360] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3451.246240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3451.259419] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3451.369440] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3455.403949] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3464.641840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3467.300119] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3803.804754] Lustre: 127754:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3803.811252] Lustre: 127754:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3803.815486] Lustre: 127754:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3803.826537] Lustre: 127754:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3803.835302] Lustre: 127754:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3803.842808] Lustre: 127754:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4044.205916] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:31:42 (1769758302) [ 4059.501482] LustreError: 127982:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4059.506082] LustreError: 127982:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4059.593950] Lustre: server umount lustre-MDT0000 complete [ 4060.647251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4060.657103] LustreError: 122156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.660294] Lustre: Skipped 3 previous similar messages [ 4060.675319] LustreError: 122156:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 4062.311654] LustreError: 127767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758322 with bad export cookie 10842203835597272839 [ 4062.313664] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4062.323641] LustreError: 127767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4065.766941] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.770358] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4065.772207] Lustre: Skipped 3 previous similar messages [ 4065.777343] Lustre: Skipped 5 previous similar messages [ 4068.635044] Lustre: server umount lustre-MDT0001 complete [ 4077.448281] Lustre: server umount lustre-OST0000 complete [ 4086.440611] Lustre: server umount lustre-OST0001 complete [ 4091.099646] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 4097.771285] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 4123.285906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4123.476916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4123.506638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4123.605982] Lustre: lustre-MDT0000: new disk, initializing [ 4123.675796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4123.680052] Lustre: Skipped 4 previous similar messages [ 4123.690750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4126.946491] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 4135.881924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4135.991277] Lustre: 130760:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4136.043958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4136.047536] Lustre: Skipped 1 previous similar message [ 4136.107404] Lustre: lustre-MDT0001: new disk, initializing [ 4136.209885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4136.217706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4139.341250] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 4145.096542] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4147.579419] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4152.789910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4153.003162] Lustre: lustre-OST0000: new disk, initializing [ 4153.009412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4154.997799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4155.011233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4155.087668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4157.887962] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 4164.059746] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4171.235489] Lustre: lustre-MDD0000: changelog on [ 4173.105544] Lustre: lustre-MDD0001: changelog on [ 5284.731673] Lustre: lustre-MDD0001: changelog off [ 5285.554076] Lustre: 130766:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5285.926608] Lustre: lustre-MDD0000: changelog off [ 5292.002490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5292.002730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.005134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5292.005140] Lustre: Skipped 2 previous similar messages [ 5292.023907] Lustre: Skipped 1 previous similar message [ 5297.122337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5297.125040] Lustre: Skipped 2 previous similar messages [ 5302.240635] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5302.247816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5302.251632] Lustre: Skipped 1 previous similar message [ 5302.277140] LustreError: 140460:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5302.280837] LustreError: 140460:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5302.355706] Lustre: server umount lustre-OST0000 complete [ 5307.362584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.365891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5307.369290] Lustre: Skipped 1 previous similar message [ 5307.374224] Lustre: Skipped 1 previous similar message [ 5310.689683] Lustre: server umount lustre-MDT0000 complete [ 5312.481211] LustreError: 130771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5312.489534] LustreError: 130771:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5312.844207] LustreError: 130751:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759572 with bad export cookie 10842203835598092658 [ 5312.847826] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5312.857662] LustreError: 130751:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5313.028665] Lustre: server umount lustre-MDT0001 complete [ 5319.177735] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 5320.901446] Key type lgssc unregistered [ 5321.126924] LNet: 141395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5321.133414] LNetError: 141395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5321.147058] LNet: Removed LNI 192.168.206.146@tcp [ 5321.728152] Key type .llcrypt unregistered [ 5321.730536] Key type ._llcrypt unregistered [ 5332.558871] Key type ._llcrypt registered [ 5332.559825] Key type .llcrypt registered [ 5332.603364] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 5340.997950] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 5341.473635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5341.515425] alg: No test for adler32 (adler32-zlib) [ 5342.483126] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 5342.606902] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 5344.224381] Key type lgssc registered [ 5344.888447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5362.730183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5362.746665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.875592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5363.891632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5363.933921] Lustre: lustre-MDT0000: new disk, initializing [ 5363.986173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.994762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5365.887447] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5371.247099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.296035] Lustre: 144533:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5371.330399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5371.332895] Lustre: Skipped 1 previous similar message [ 5371.370671] Lustre: lustre-MDT0001: new disk, initializing [ 5371.410297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5371.427561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5371.431696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5373.151688] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5377.257465] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5379.025090] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5382.443308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5382.586788] Lustre: lustre-OST0000: new disk, initializing [ 5382.589208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5382.630215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.794220] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5388.813369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5388.823626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5388.917770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5390.052041] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5394.011662] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_module obdecho/obdecho [ 5394.857895] LustreError: 147147:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5394.861068] LustreError: 147147:0:(obd_config.c:859:class_setup()) setup ec failed (-22) [ 5395.755064] LustreError: 147243:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 5402.082111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5402.085792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5402.091234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5404.132693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.133346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5404.137786] Lustre: Skipped 1 previous similar message [ 5404.140113] Lustre: Skipped 1 previous similar message [ 5406.681664] LustreError: 147583:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5406.751399] Lustre: server umount lustre-MDT0000 complete [ 5408.866724] LustreError: 146273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759668 with bad export cookie 17435659634458087697 [ 5408.871261] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5408.873314] LustreError: 146273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5408.954994] LustreError: 147784:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5408.959061] LustreError: 147784:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5409.096902] Lustre: server umount lustre-MDT0001 complete [ 5420.157050] LustreError: 147984:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5420.160133] LustreError: 147984:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5420.238499] Lustre: server umount lustre-OST0000 complete [ 5431.705420] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 5436.622996] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 5455.136752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.231847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5455.247308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5455.290624] Lustre: lustre-MDT0000: new disk, initializing [ 5455.331560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5455.338679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5457.188633] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5463.033131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5463.090146] Lustre: 150915:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5463.120806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5463.123154] Lustre: Skipped 1 previous similar message [ 5463.162088] Lustre: lustre-MDT0001: new disk, initializing [ 5463.203756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5463.228854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5463.232096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5465.135388] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5469.405559] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5471.124726] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5474.551862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5474.693398] Lustre: lustre-OST0000: new disk, initializing [ 5474.696531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5474.754914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5477.756573] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5482.512659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5482.516948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5482.591490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5482.623709] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5484.847364] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5484.970600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5487.597771] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5487.768992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5492.708333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5492.709324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5492.711669] LustreError: Skipped 1 previous similar message [ 5492.712689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5492.717209] Lustre: Skipped 1 previous similar message [ 5492.727441] Lustre: Skipped 1 previous similar message [ 5495.162183] LustreError: 153845:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5495.167960] LustreError: 153845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5495.236253] Lustre: server umount lustre-OST0000 complete [ 5497.825415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.832447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5497.835260] Lustre: Skipped 1 previous similar message [ 5502.947675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5502.952975] Lustre: Skipped 2 previous similar messages [ 5503.337549] LustreError: 154046:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5503.341154] LustreError: 154046:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5503.406743] Lustre: server umount lustre-MDT0000 complete [ 5505.539570] LustreError: 150908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759765 with bad export cookie 17435659634458088642 [ 5505.544167] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5505.546026] LustreError: 150908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5505.678969] Lustre: server umount lustre-MDT0001 complete [ 5507.360338] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:56:06 (1769759766) [ 5508.229148] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5509.244106] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:56:08 (1769759768) [ 5512.862316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.112150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.220569] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5518.309542] LustreError: 154916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.314853] LustreError: 154916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5519.714676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.420966] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5524.938292] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5526.494381] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5529.612452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5532.356776] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5534.857044] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5549.546803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.556314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5549.560117] Lustre: Skipped 2 previous similar messages [ 5549.571610] Lustre: Skipped 2 previous similar messages [ 5553.936205] LustreError: 157178:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5553.938961] LustreError: 157178:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5553.969170] Lustre: server umount lustre-MDT0000 complete [ 5554.657583] LustreError: 155933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.667087] LustreError: 155933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5555.675439] LustreError: 156531:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759815 with bad export cookie 17435659634458089524 [ 5555.678627] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5555.682786] LustreError: 156531:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5555.744248] LustreError: 155626:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004cd62fc0 x1855727444517504/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5559.777747] LustreError: 154912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5559.778726] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.787771] LustreError: 154912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5561.861413] Lustre: server umount lustre-MDT0001 complete [ 5565.210873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5571.768115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5571.892896] LustreError: 158441:0:(llog.c:1643:llog_backup()) MGC192.168.206.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5571.896243] Lustre: 158441:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.206.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5581.075381] LustreError: 158448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.080729] LustreError: 158448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5585.384714] LustreError: 158447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5585.393856] LustreError: 158447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5595.412746] LustreError: 158448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.423305] LustreError: 158448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5611.796836] LustreError: 158448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5611.804160] LustreError: 158448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 5632.992648] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5633.000074] Lustre: 158441:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.206.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5646.305200] LustreError: 158448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5646.314594] LustreError: 158448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 5658.592301] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 5658.594733] Lustre: 154907:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.206.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5673.450330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5673.453435] Lustre: Skipped 2 previous similar messages [ 5675.005966] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5675.490828] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5678.290968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.624208] LustreError: 154907:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5700.065809] LustreError: 142758:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000094174f70 x1855727444535296/t0(0) o250->MGC192.168.206.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5700.079125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 5701.729151] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5703.781259] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5704.465354] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5705.701191] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5705.890351] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5708.596556] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 5715.938316] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.939709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5715.940183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5715.940191] Lustre: Skipped 4 previous similar messages [ 5715.948244] Lustre: Skipped 2 previous similar messages [ 5720.964710] LustreError: 159929:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5720.967102] LustreError: 159929:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5720.997067] Lustre: server umount lustre-MDT0000 complete [ 5721.060089] LustreError: 159838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.067687] LustreError: 159838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 44 previous similar messages [ 5722.599219] LustreError: 154895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759982 with bad export cookie 17435659634458095999 [ 5722.602640] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5722.603779] LustreError: 154895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5722.755040] Lustre: server umount lustre-MDT0001 complete [ 5734.661408] Lustre: server umount lustre-OST0000 complete [ 5742.899644] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 5746.724207] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 5761.685365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.800432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5761.814621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5761.856205] Lustre: lustre-MDT0000: new disk, initializing [ 5761.890338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5761.894487] Lustre: Skipped 1 previous similar message [ 5761.904272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5763.425561] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5768.503508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.548562] Lustre: 163258:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5768.681756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5770.250527] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5773.699183] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5775.103725] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5777.732809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5780.537420] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5784.047353] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5785.666202] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5787.645503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5787.695775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5787.836504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5789.344909] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5789.420736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5792.738065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5792.738829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5792.741048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5792.741053] Lustre: Skipped 2 previous similar messages [ 5792.741147] LustreError: Skipped 1 previous similar message [ 5792.751022] Lustre: Skipped 1 previous similar message [ 5796.212878] LustreError: 166202:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5796.217572] LustreError: 166202:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5796.282181] Lustre: server umount lustre-OST0000 complete [ 5812.192161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5812.279238] Lustre: server umount lustre-MDT0000 complete [ 5813.863333] LustreError: 163253:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760073 with bad export cookie 17435659634458112827 [ 5813.864116] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5813.867991] LustreError: 163253:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5814.007573] Lustre: server umount lustre-MDT0001 complete [ 5817.861164] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:01:17 (1769760077) [ 5820.829220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.844325] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5827.224794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5829.120562] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5831.747137] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5833.480789] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5836.435439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5838.866894] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5841.118602] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5856.636872] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:01:55 (1769760115) [ 5858.273145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.276188] LustreError: Skipped 1 previous similar message [ 5858.278171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.283253] Lustre: Skipped 2 previous similar messages [ 5858.285667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.287755] Lustre: Skipped 8 previous similar messages [ 5863.858860] LustreError: 169746:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5863.863053] LustreError: 169746:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5863.913946] Lustre: server umount lustre-OST0000 complete [ 5880.288097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5880.352316] Lustre: server umount lustre-MDT0000 complete [ 5881.734077] LustreError: 167266:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760141 with bad export cookie 17435659634458113695 [ 5881.736351] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5881.737510] LustreError: 167266:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5881.853982] Lustre: server umount lustre-MDT0001 complete [ 5885.749839] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 5886.909569] Key type lgssc unregistered [ 5887.088140] LNet: 170681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5887.092936] LNetError: 170681:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5887.108227] LNet: Removed LNI 192.168.206.146@tcp [ 5887.580336] Key type .llcrypt unregistered [ 5887.584318] Key type ._llcrypt unregistered [ 5896.336256] Key type ._llcrypt registered [ 5896.337394] Key type .llcrypt registered [ 5896.389914] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 5896.758538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5896.768449] alg: No test for adler32 (adler32-zlib) [ 5897.644860] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 5897.740698] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 5899.320158] Key type lgssc registered [ 5899.742038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5903.211311] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5903.217974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5918.688565] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5923.808194] LustreError: 171882:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.146@tcp: failed processing log, type 4: rc = -110 [ 5928.929127] LustreError: 171882:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5928.935432] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5928.938210] LustreError: MGC192.168.206.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5928.943291] LustreError: 171882:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5928.947134] LustreError: 171882:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -22 [ 5928.952741] LustreError: 171882:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 5928.956630] LustreError: 171882:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 5929.017534] Lustre: server umount lustre-OST0000 complete [ 5929.020935] LustreError: 171882:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5932.783501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5933.973590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5935.547717] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5938.893153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5939.048147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5940.635785] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5943.221537] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5944.868877] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5948.042469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5948.125028] LustreError: 173965:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5948.128380] Lustre: 173965:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.206.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5948.196530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.731753] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 5953.209943] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5954.227143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5954.228407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5954.231348] Lustre: Skipped 2 previous similar messages [ 5954.237687] Lustre: Skipped 2 previous similar messages [ 5959.651585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5959.654303] Lustre: Skipped 2 previous similar messages [ 5960.317325] LustreError: 174615:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5960.319088] LustreError: 174615:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5960.351602] Lustre: server umount lustre-MDT0000 complete [ 5961.886457] LustreError: 172360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760221 with bad export cookie 10938313953878655763 [ 5961.887317] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5961.891888] LustreError: 172360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5961.962521] LustreError: 174819:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5961.964864] LustreError: 174819:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5962.040819] Lustre: server umount lustre-MDT0001 complete [ 5973.939629] LustreError: 175019:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5973.941742] LustreError: 175019:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5974.001315] Lustre: server umount lustre-OST0000 complete [ 5977.330195] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 5980.807477] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 5995.421785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.555115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5995.568072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5995.604171] Lustre: lustre-MDT0000: new disk, initializing [ 5995.630846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5995.640184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5997.204656] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6002.322604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6002.373870] Lustre: 177342:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6002.402118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6002.405208] Lustre: Skipped 1 previous similar message [ 6002.441717] Lustre: lustre-MDT0001: new disk, initializing [ 6002.478720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6002.492799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6002.499344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6004.000491] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6007.648551] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6009.097252] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6012.059582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6012.202759] LustreError: 179066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6012.210167] LustreError: 179066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6012.221088] Lustre: lustre-OST0000: new disk, initializing [ 6012.224524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6012.259620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6014.689289] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6017.548195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6017.551517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6017.578387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 6018.671314] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6020.634304] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6020.729477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6022.248627] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6022.342690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6023.650516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.651137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6023.651667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6023.653818] Lustre: Skipped 1 previous similar message [ 6025.199053] LustreError: 180269:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 6025.201208] LustreError: 180269:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6025.238905] Lustre: server umount lustre-OST0000 complete [ 6028.256819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6028.260204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.265820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6028.268270] Lustre: Skipped 1 previous similar message [ 6032.677257] Lustre: server umount lustre-MDT0000 complete [ 6033.888991] LustreError: 177353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.893906] LustreError: 177353:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6034.117200] LustreError: 177333:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760294 with bad export cookie 10938313953878656463 [ 6034.119816] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6034.123683] LustreError: 177333:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6034.193518] LustreError: 180671:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6034.196546] LustreError: 180671:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6034.257362] Lustre: server umount lustre-MDT0001 complete [ 6037.734334] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:04:56 (1769760296) [ 6045.437624] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 6046.749930] Key type lgssc unregistered [ 6046.910734] LNet: 181854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6046.914836] LNetError: 181854:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6046.922685] LNet: Removed LNI 192.168.206.146@tcp [ 6047.302164] Key type .llcrypt unregistered [ 6047.303322] Key type ._llcrypt unregistered [ 6055.916731] Key type ._llcrypt registered [ 6055.918062] Key type .llcrypt registered [ 6055.958954] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 6056.410396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6056.416905] alg: No test for adler32 (adler32-zlib) [ 6057.289194] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 6057.400682] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 6058.992120] Key type lgssc registered [ 6059.484611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6062.609307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6062.619441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.817583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6065.399913] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6069.497349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.660696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6071.455523] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6073.706972] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6075.046378] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6078.359480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6078.464165] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6078.466409] LustreError: 184662:0:(llog.c:1684:llog_backup()) MGC192.168.206.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6078.470196] Lustre: 184662:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.206.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6078.520208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6080.730512] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6082.806647] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6084.579632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.586603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6084.587927] Lustre: Skipped 2 previous similar messages [ 6084.589992] Lustre: Skipped 1 previous similar message [ 6089.697987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6089.700247] Lustre: Skipped 2 previous similar messages [ 6089.787731] LustreError: 185312:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6089.813115] Lustre: server umount lustre-MDT0000 complete [ 6091.209226] LustreError: 183009:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760351 with bad export cookie 10832573156706354220 [ 6091.210334] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6091.219667] LustreError: 183009:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6091.286384] LustreError: 185514:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6091.288245] LustreError: 185514:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6091.348702] Lustre: server umount lustre-MDT0001 complete [ 6102.065599] LustreError: 185714:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6102.067754] LustreError: 185714:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6102.107706] Lustre: server umount lustre-OST0000 complete [ 6105.416121] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 6109.051315] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 6122.822184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6122.915666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6122.935884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6122.977555] Lustre: lustre-MDT0000: new disk, initializing [ 6123.005637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6123.013352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6124.693919] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6129.554918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6129.600066] Lustre: 188037:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6129.622071] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6129.623607] Lustre: Skipped 1 previous similar message [ 6129.625366] LustreError: 188800:0:(llog.c:1684:llog_backup()) MGC192.168.206.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6129.628895] LustreError: 188800:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 6129.632073] Lustre: 188800:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.206.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6129.643740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6129.645954] Lustre: Skipped 1 previous similar message [ 6129.683793] Lustre: lustre-MDT0001: new disk, initializing [ 6129.722958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6129.742803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6129.746593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6131.494227] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6134.964316] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6136.301464] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6138.670561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6138.764852] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6138.766315] Lustre: Skipped 1 previous similar message [ 6138.767633] LustreError: 189754:0:(llog.c:1684:llog_backup()) MGC192.168.206.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6138.770522] LustreError: 189754:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 6138.773131] Lustre: 189754:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.206.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6138.776040] Lustre: 189754:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6138.784583] Lustre: lustre-OST0000: new disk, initializing [ 6138.786526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6138.814095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6140.807657] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6142.969467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6142.972548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6143.006563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6143.923976] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6145.295754] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6145.375187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6146.853328] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6146.949447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6148.065609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6148.065686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6148.069304] LustreError: Skipped 1 previous similar message [ 6148.074474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6148.076749] Lustre: Skipped 1 previous similar message [ 6149.742541] LustreError: 190968:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6149.745118] LustreError: 190968:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6149.787873] Lustre: server umount lustre-OST0000 complete [ 6153.185456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6153.187831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6153.190452] Lustre: Skipped 2 previous similar messages [ 6157.313873] LustreError: 191169:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6157.316274] LustreError: 191169:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6157.348325] Lustre: server umount lustre-MDT0000 complete [ 6158.304969] LustreError: 188048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.312121] LustreError: 188048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6158.801697] LustreError: 188813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760418 with bad export cookie 10832573156706354920 [ 6158.804970] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6158.807433] LustreError: 188813:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6158.919989] Lustre: server umount lustre-MDT0001 complete [ 6162.038885] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:07:01 (1769760421) [ 6167.853442] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 6171.785700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.927894] LustreError: 192734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6171.943725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6173.321337] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6176.530439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.210304] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6179.476811] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6181.968557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6184.393190] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6187.964590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6188.049842] Lustre: lustre-OST0001: new disk, initializing [ 6188.052110] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6190.548407] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6196.238558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6196.244595] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6196.285893] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6200.905361] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6210.049292] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6215.760349] Lustre: DEBUG MARKER: ADD OST3 [ 6217.642988] loop: module loaded [ 6217.663245] loop0: detected capacity change from 0 to 204800000 [ 6217.755745] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6217.763198] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6217.770946] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6219.952812] loop0: detected capacity change from 0 to 204800000 [ 6221.979117] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6222.051944] Lustre: lustre-OST0003: new disk, initializing [ 6222.054766] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6222.096745] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6222.100022] Lustre: Skipped 3 previous similar messages [ 6222.828169] Lustre: DEBUG MARKER: STOP OST3 [ 6223.329826] Lustre: *** cfs_fail_loc=2109, val=2*** [ 6223.376267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6223.380476] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6223.394450] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 6223.608181] Lustre: Failing over lustre-OST0003 [ 6223.634096] LustreError: 197041:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 6223.637524] LustreError: 197041:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6223.661957] LustreError: 195505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6223.662480] Lustre: server umount lustre-OST0003 complete [ 6224.476361] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6225.377222] LustreError: 196986:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 6228.454232] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6228.454748] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.457469] LustreError: 194185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.461622] Lustre: Skipped 1 previous similar message [ 6228.743570] Lustre: DEBUG MARKER: START OST3 again [ 6231.126551] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6231.219426] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6232.431668] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6232.440615] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6232.440616] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6233.249855] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6246.951767] Lustre: Failing over lustre-OST0003 [ 6246.979075] LustreError: 199008:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 6246.982151] LustreError: 199008:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6247.005626] Lustre: server umount lustre-OST0003 complete [ 6247.905687] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6247.906951] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6247.912714] LustreError: Skipped 1 previous similar message [ 6247.917100] LustreError: 195505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.923996] LustreError: 195505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6260.196286] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6260.198332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6260.198694] LustreError: 194186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6260.198705] LustreError: 194186:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6260.200969] Lustre: Skipped 4 previous similar messages [ 6260.213245] Lustre: Skipped 4 previous similar messages [ 6266.147403] Lustre: server umount lustre-MDT0000 complete [ 6267.541099] LustreError: 192716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760527 with bad export cookie 10832573156706355802 [ 6267.544610] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.545417] LustreError: 192716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6270.439356] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6270.442713] Lustre: Skipped 5 previous similar messages [ 6273.789763] Lustre: server umount lustre-MDT0001 complete [ 6275.325833] Lustre: server umount lustre-OST0000 complete [ 6276.820031] Lustre: server umount lustre-OST0001 complete [ 6279.289631] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 6282.099635] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 6294.699987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6294.788135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6294.800094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6294.836485] Lustre: lustre-MDT0000: new disk, initializing [ 6294.868564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6294.870700] Lustre: Skipped 1 previous similar message [ 6294.876951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6296.206838] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6300.693177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6300.735240] Lustre: 202050:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6300.833896] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6302.174611] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6305.274531] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6306.457258] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6308.767030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6310.139805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6310.176447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6310.847654] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6314.229818] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6315.839479] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6315.915393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6317.355189] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6317.437342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6320.174077] LustreError: 204981:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6320.176816] LustreError: 204981:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 6320.223459] Lustre: server umount lustre-OST0000 complete [ 6321.632891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6321.635731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6321.640133] Lustre: Skipped 2 previous similar messages [ 6321.642496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6321.644687] Lustre: Skipped 2 previous similar messages [ 6331.901940] Lustre: server umount lustre-MDT0000 complete [ 6333.310631] LustreError: 202041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760593 with bad export cookie 10832573156706358077 [ 6333.313556] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.316881] LustreError: 202041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6343.703800] Lustre: server umount lustre-MDT0001 complete [ 6345.011367] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:10:04 (1769760604) [ 6350.357991] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 6353.623890] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 6366.674762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.791310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6366.807029] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6366.809329] Lustre: Skipped 3 previous similar messages [ 6366.847792] Lustre: lustre-MDT0000: new disk, initializing [ 6366.850136] Lustre: Skipped 2 previous similar messages [ 6366.909694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6366.914350] Lustre: Skipped 2 previous similar messages [ 6368.594725] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6373.603241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.670972] Lustre: 208211:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6373.790354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6375.397165] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6378.738360] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6380.012732] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6382.461421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6384.104629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6384.888767] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6388.143568] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6389.622957] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6389.699890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6391.169841] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6391.260538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6394.338565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6394.341025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6394.342141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6394.344675] Lustre: Skipped 2 previous similar messages [ 6394.346641] Lustre: Skipped 4 previous similar messages [ 6398.000201] LustreError: 211141:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6398.003075] LustreError: 211141:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6398.049832] Lustre: server umount lustre-OST0000 complete [ 6405.623439] Lustre: server umount lustre-MDT0000 complete [ 6407.254951] LustreError: 208204:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760667 with bad export cookie 10832573156706358959 [ 6407.255968] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6407.259926] LustreError: 208204:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6407.387221] Lustre: server umount lustre-MDT0001 complete [ 6410.444201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6410.598806] LustreError: 212018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6410.605479] LustreError: 212018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6412.114211] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6415.616879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6417.310086] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6419.329804] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6420.754441] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6423.483663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6423.622564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6423.625295] Lustre: Skipped 7 previous similar messages [ 6425.718417] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6427.556219] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6446.563349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6446.566414] LustreError: Skipped 3 previous similar messages [ 6450.453531] Lustre: server umount lustre-OST0000 complete [ 6461.921296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6461.923630] Lustre: Skipped 9 previous similar messages [ 6466.016440] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6466.079259] Lustre: server umount lustre-MDT0000 complete [ 6467.371459] LustreError: 212043:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760727 with bad export cookie 10832573156706359841 [ 6467.374985] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6467.377553] LustreError: 212043:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6467.497970] Lustre: server umount lustre-MDT0001 complete [ 6561.249730] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 6562.130959] Key type lgssc unregistered [ 6562.242338] LNet: 215348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6562.244576] LNetError: 215348:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6562.253256] LNet: Removed LNI 192.168.206.146@tcp [ 6562.534103] Key type .llcrypt unregistered [ 6562.535125] Key type ._llcrypt unregistered [ 6570.854549] Key type ._llcrypt registered [ 6570.855755] Key type .llcrypt registered [ 6570.897086] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 6577.251546] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 6577.604090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6577.678078] alg: No test for adler32 (adler32-zlib) [ 6578.533377] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 6578.621751] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 6580.208144] Key type lgssc registered [ 6580.598905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6592.371462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6592.376017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.469168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6593.481727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6593.516659] Lustre: lustre-MDT0000: new disk, initializing [ 6593.542580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6593.551609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6594.861852] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6599.142910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6599.182805] Lustre: 218636:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6599.205994] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6599.208021] Lustre: Skipped 1 previous similar message [ 6599.245623] Lustre: lustre-MDT0001: new disk, initializing [ 6599.282529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6599.296697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6599.301126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6600.617244] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6603.707311] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6604.850060] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6607.076843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6607.169304] Lustre: lustre-OST0000: new disk, initializing [ 6607.171752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6607.200139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.078685] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6609.334929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6609.338480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6609.374736] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6612.217213] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6613.593404] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6613.673667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6615.038049] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6615.111441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6619.617972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6619.619032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.620766] LustreError: Skipped 1 previous similar message [ 6619.621512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6619.624103] Lustre: Skipped 1 previous similar message [ 6619.628047] Lustre: Skipped 1 previous similar message [ 6624.740084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6624.742682] Lustre: Skipped 1 previous similar message [ 6629.859480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6629.862234] Lustre: Skipped 1 previous similar message [ 6630.368218] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6630.386186] LustreError: 221567:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6630.435314] Lustre: server umount lustre-OST0000 complete [ 6634.977912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.978433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6634.982192] Lustre: Skipped 1 previous similar message [ 6637.820566] LustreError: 221772:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6637.823050] LustreError: 221772:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6637.850376] Lustre: server umount lustre-MDT0000 complete [ 6639.125142] LustreError: 218628:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760899 with bad export cookie 15712905132073979075 [ 6639.128768] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6639.129219] LustreError: 218628:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6639.189732] LustreError: 221973:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6639.191711] LustreError: 221973:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6639.246766] Lustre: server umount lustre-MDT0001 complete [ 6644.988341] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 6648.394063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6648.576713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6649.968947] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6652.773774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.897419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6654.347983] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6655.384261] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6657.822790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6659.913256] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6662.992320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6663.069159] Lustre: lustre-OST0001: new disk, initializing [ 6663.071382] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6663.104649] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6663.106683] Lustre: Skipped 1 previous similar message [ 6663.349623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6663.353793] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6663.382975] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6665.380676] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6671.058664] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6676.745958] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6685.860302] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:15:45 (1769760945) [ 6688.737021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6688.739659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6688.743892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6688.745934] Lustre: Skipped 1 previous similar message [ 6692.607818] LustreError: 226514:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6692.610213] LustreError: 226514:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6692.644734] Lustre: server umount lustre-MDT0000 complete [ 6693.911074] LustreError: 223168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760953 with bad export cookie 15712905132073979943 [ 6693.914518] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6693.915633] LustreError: 223168:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6694.020806] Lustre: server umount lustre-MDT0001 complete [ 6705.526631] LustreError: 226915:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6705.528956] LustreError: 226915:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6705.547070] Lustre: server umount lustre-OST0000 complete [ 6716.981262] LustreError: 227117:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6716.983835] LustreError: 227117:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6717.032518] Lustre: server umount lustre-OST0001 complete [ 6719.071736] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 6722.226574] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 6733.946120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6734.039684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6734.053658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6734.089249] Lustre: lustre-MDT0000: new disk, initializing [ 6734.113169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6734.119341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6735.416635] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6739.720180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.762225] Lustre: 229290:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6739.880059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6741.258511] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6744.497682] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6745.895931] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6748.387234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6748.499503] Lustre: lustre-OST0000: new disk, initializing [ 6748.501514] Lustre: Skipped 1 previous similar message [ 6748.503612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6748.506208] Lustre: Skipped 2 previous similar messages [ 6750.215162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6750.218231] Lustre: Skipped 1 previous similar message [ 6750.219854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6750.251373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6750.766746] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6753.991121] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6755.446264] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6755.533291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6756.919856] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6757.011237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6760.419307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.421116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6760.425670] Lustre: Skipped 3 previous similar messages [ 6760.426426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6760.428474] LustreError: Skipped 1 previous similar message [ 6760.435569] Lustre: Skipped 5 previous similar messages [ 6772.193049] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6772.214754] LustreError: 232219:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6772.217016] LustreError: 232219:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6772.265852] Lustre: server umount lustre-OST0000 complete [ 6775.778411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.783095] Lustre: Skipped 2 previous similar messages [ 6775.784783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6779.815181] Lustre: server umount lustre-MDT0000 complete [ 6780.896822] LustreError: 230287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6780.902220] LustreError: 230287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6781.290532] LustreError: 229282:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761041 with bad export cookie 15712905132073982043 [ 6781.295798] LustreError: 229282:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6781.297397] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6781.427743] Lustre: server umount lustre-MDT0001 complete [ 6784.543167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6784.718294] LustreError: 233098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.748971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6784.751794] Lustre: Skipped 2 previous similar messages [ 6786.145408] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6789.172576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6790.723495] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6792.781147] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6794.105341] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6796.459651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6798.971507] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6800.900216] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6948.324127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6948.324799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6948.331639] LustreError: Skipped 1 previous similar message [ 6948.336671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6948.336676] Lustre: Skipped 6 previous similar messages [ 6949.552110] LustreError: 235379:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6949.554086] LustreError: 235379:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6949.596705] Lustre: server umount lustre-OST0000 complete [ 6965.216103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6965.276109] Lustre: server umount lustre-MDT0000 complete [ 6966.599334] LustreError: 233078:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761226 with bad export cookie 15712905132073982925 [ 6966.602521] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6966.603163] LustreError: 233078:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6966.726259] Lustre: server umount lustre-MDT0001 complete [ 7064.721799] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 7065.754815] Key type lgssc unregistered [ 7065.884346] LNet: 236350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7065.886981] LNetError: 236350:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7065.893350] LNet: Removed LNI 192.168.206.146@tcp [ 7066.199103] Key type .llcrypt unregistered [ 7066.200118] Key type ._llcrypt unregistered [ 7074.530625] Key type ._llcrypt registered [ 7074.531633] Key type .llcrypt registered [ 7074.568801] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 7080.811192] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7081.131427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7081.169031] alg: No test for adler32 (adler32-zlib) [ 7082.023084] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7082.109958] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 7083.696097] Key type lgssc registered [ 7084.078045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7098.999913] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7102.445491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7102.451602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.539824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7103.552762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7103.587721] Lustre: lustre-MDT0000: new disk, initializing [ 7103.613638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7103.620991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7104.865084] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7109.543191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.589485] Lustre: 240335:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7109.622410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7109.625523] Lustre: Skipped 1 previous similar message [ 7109.670188] Lustre: lustre-MDT0001: new disk, initializing [ 7109.705156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7109.721515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7109.726161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7111.173736] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7113.456285] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7115.649062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7115.751858] Lustre: lustre-OST0000: new disk, initializing [ 7115.753967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7115.781444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7117.668110] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7118.718328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7118.722469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7118.764517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7121.797639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7121.862248] Lustre: lustre-OST0001: new disk, initializing [ 7121.864461] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7121.899533] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7123.742195] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7127.045768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7127.048978] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7127.081469] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7129.243445] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7133.902972] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7142.372070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7142.373324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7142.376164] Lustre: Skipped 3 previous similar messages [ 7142.379900] Lustre: Skipped 3 previous similar messages [ 7147.489130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7148.105715] LustreError: 243847:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7148.133253] Lustre: server umount lustre-MDT0000 complete [ 7149.391408] LustreError: 241113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761409 with bad export cookie 824180689635486815 [ 7149.393044] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7149.396126] LustreError: 241113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7149.459231] LustreError: 244049:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7149.462116] LustreError: 244049:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7149.515848] Lustre: server umount lustre-MDT0001 complete [ 7160.826091] LustreError: 244249:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7160.828635] LustreError: 244249:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7160.849959] Lustre: server umount lustre-OST0000 complete [ 7172.406709] LustreError: 244451:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7172.408863] LustreError: 244451:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7172.455463] Lustre: server umount lustre-OST0001 complete [ 7177.115951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7180.413597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7183.358185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7186.841864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7193.622715] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7197.207430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7197.214260] Lustre: lustre-MDT0000: reset Object Index mappings [ 7197.390252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7198.768192] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7201.731931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.737446] Lustre: lustre-MDT0001: reset Object Index mappings [ 7203.250374] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7204.233867] Lustre: 248861:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7206.416884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7206.541550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7206.543974] Lustre: Skipped 1 previous similar message [ 7207.588632] LustreError: 249216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7207.591363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7207.594857] LustreError: 249216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7208.354193] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7211.281192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7212.402706] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7213.291988] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7216.348114] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7217.527398] Lustre: 250701:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7227.360879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7227.363283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.367220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7227.368450] Lustre: Skipped 3 previous similar messages [ 7229.757458] LustreError: 250805:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7229.759722] LustreError: 250805:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7229.789764] Lustre: server umount lustre-MDT0000 complete [ 7230.884981] LustreError: 250703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761490 with bad export cookie 824180689635493598 [ 7230.888103] LustreError: 250703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7230.888303] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7230.988734] Lustre: server umount lustre-MDT0001 complete [ 7242.163104] LustreError: 251207:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7242.165564] LustreError: 251207:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 7242.181755] Lustre: server umount lustre-OST0000 complete [ 7253.665654] Lustre: server umount lustre-OST0001 complete [ 7259.854135] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:25:19 (1769761519) [ 7265.002377] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 7267.995861] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7279.573195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.657083] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7279.668464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7279.707384] Lustre: lustre-MDT0000: new disk, initializing [ 7279.730934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7279.732953] Lustre: Skipped 1 previous similar message [ 7279.738239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7281.087423] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7285.243080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7285.284431] Lustre: 254915:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7285.389844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7286.750278] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7289.962667] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7291.199359] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7293.489884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7293.601415] Lustre: lustre-OST0000: new disk, initializing [ 7293.602960] Lustre: Skipped 1 previous similar message [ 7293.605508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7293.610360] Lustre: Skipped 2 previous similar messages [ 7295.555975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7295.558587] Lustre: Skipped 1 previous similar message [ 7295.560292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7295.602341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7295.604751] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7298.589480] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7299.909905] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7299.974522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7301.233468] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7301.304695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7305.696766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7305.698627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7305.701765] Lustre: Skipped 3 previous similar messages [ 7305.703571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7305.705390] Lustre: Skipped 3 previous similar messages [ 7308.013080] LustreError: 257846:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7308.015418] LustreError: 257846:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7308.050039] Lustre: server umount lustre-OST0000 complete [ 7311.330215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.332515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7311.334199] Lustre: Skipped 1 previous similar message [ 7311.336696] LustreError: Skipped 1 previous similar message [ 7311.337331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.341336] Lustre: Skipped 3 previous similar messages [ 7315.544459] Lustre: server umount lustre-MDT0000 complete [ 7316.449327] LustreError: 256665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7316.455432] LustreError: 256665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7316.765059] LustreError: 254907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761576 with bad export cookie 824180689635495691 [ 7316.768171] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7316.770065] LustreError: 254907:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7316.878904] Lustre: server umount lustre-MDT0001 complete [ 7322.391473] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7325.612297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.736135] LustreError: 259418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7325.750025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7325.751882] Lustre: Skipped 2 previous similar messages [ 7327.011813] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7329.794707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.102974] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7332.050731] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7334.226700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7336.179416] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7338.867076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7338.930887] Lustre: lustre-OST0001: new disk, initializing [ 7338.932513] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7340.744543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7340.751408] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7340.785855] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7340.885651] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7345.257341] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7346.478323] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7353.251029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7353.254629] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7355.872805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7355.874641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7355.878191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7361.279653] LustreError: 262915:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7361.281696] LustreError: 262915:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7361.311355] Lustre: server umount lustre-MDT0000 complete [ 7361.505394] LustreError: 260127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7361.510216] LustreError: 260127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7362.504205] LustreError: 260135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761622 with bad export cookie 824180689635496573 [ 7362.507283] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7362.507939] LustreError: 260135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7362.623062] Lustre: server umount lustre-MDT0001 complete [ 7373.828051] Lustre: server umount lustre-OST0000 complete [ 7385.301664] Lustre: server umount lustre-OST0001 complete [ 7390.275474] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7393.572808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.696658] LustreError: 264693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7393.713332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7393.715041] Lustre: Skipped 3 previous similar messages [ 7394.909381] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7397.684566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7399.085293] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7400.033944] Lustre: 265796:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7402.165540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7403.301502] LustreError: 266151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7403.307127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7404.099772] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7406.952413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7408.044293] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7408.792885] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7411.887812] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7413.107670] Lustre: 267646:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7419.819965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7419.850781] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7422.856761] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:28:02 (1769761682) [ 7438.824557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7438.825163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.828294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7438.830863] Lustre: Skipped 5 previous similar messages [ 7438.833214] Lustre: Skipped 5 previous similar messages [ 7440.446118] LustreError: 268355:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7440.449326] LustreError: 268355:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 7440.486808] Lustre: server umount lustre-MDT0000 complete [ 7441.701183] LustreError: 264674:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761701 with bad export cookie 824180689635502866 [ 7441.702705] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7441.704780] LustreError: 264674:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7443.937602] LustreError: 266172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.943115] LustreError: 266172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7456.224098] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7456.303922] Lustre: server umount lustre-MDT0001 complete [ 7463.619919] Lustre: server umount lustre-OST0000 complete [ 7479.264105] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7479.324609] Lustre: server umount lustre-OST0001 complete [ 7481.175389] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 7484.052737] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7498.131854] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7501.464554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7501.542569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7501.552787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7501.587732] Lustre: lustre-MDT0000: new disk, initializing [ 7501.615341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7502.931425] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7506.830331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7506.867873] Lustre: 271827:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7506.958270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7508.188723] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7512.399439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.773239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7514.217121] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7518.122946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7519.275189] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7520.069961] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7524.365268] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7534.064720] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 03:29:53 (1769761793) [ 7537.632962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7537.636405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7537.641191] Lustre: Skipped 3 previous similar messages [ 7537.643511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7537.645596] Lustre: Skipped 9 previous similar messages [ 7540.827120] Lustre: server umount lustre-MDT0000 complete [ 7542.087177] LustreError: 271820:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761802 with bad export cookie 824180689635510748 [ 7542.091090] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7542.091814] LustreError: 271820:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7542.209978] Lustre: server umount lustre-MDT0001 complete [ 7553.990667] Lustre: server umount lustre-OST0000 complete [ 7565.536700] Lustre: server umount lustre-OST0001 complete [ 7571.406811] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 7574.460087] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7586.006552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7586.091148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7586.093766] Lustre: Skipped 2 previous similar messages [ 7586.101900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7586.103381] Lustre: Skipped 4 previous similar messages [ 7586.133638] Lustre: lustre-MDT0000: new disk, initializing [ 7586.134843] Lustre: Skipped 3 previous similar messages [ 7586.157626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.160013] Lustre: Skipped 7 previous similar messages [ 7586.166357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7586.170123] Lustre: Skipped 3 previous similar messages [ 7587.367992] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7591.363415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7591.402405] Lustre: 278861:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7591.502276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7591.505149] Lustre: Skipped 2 previous similar messages [ 7592.726962] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7595.699769] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7596.815157] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7598.916577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7600.663878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7600.777288] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7603.623435] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7604.302268] Lustre: Mounted lustre-client [ 7604.595368] LustreError: 281355:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8f04cec8c800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 7604.599868] LustreError: 281355:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7604.602141] LustreError: 281355:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 7604.617409] Lustre: Unmounted lustre-client [ 7609.315396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7609.316188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7609.319302] Lustre: Skipped 4 previous similar messages [ 7609.321835] Lustre: Skipped 4 previous similar messages [ 7613.325172] Lustre: server umount lustre-OST0000 complete [ 7616.992693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7620.637721] Lustre: server umount lustre-MDT0000 complete [ 7621.788376] LustreError: 278853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761881 with bad export cookie 824180689635512274 [ 7621.791422] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7621.792684] LustreError: 278853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7621.903911] Lustre: server umount lustre-MDT0001 complete [ 7625.314937] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 7626.273165] Key type lgssc unregistered [ 7660.704174] LNet: 1 local NIs in recovery (showing 1): 192.168.206.146@tcp [ 7661.216430] LNet: 282630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7661.218279] LNetError: 282630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7734.950291] LNet: Removed LNI 192.168.206.146@tcp [ 7735.224098] Key type .llcrypt unregistered [ 7735.224904] Key type ._llcrypt unregistered [ 7740.587293] Key type ._llcrypt registered [ 7740.588182] Key type .llcrypt registered [ 7740.626327] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 7746.647116] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7746.941183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7746.951933] alg: No test for adler32 (adler32-zlib) [ 7747.804856] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7747.892419] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 7749.464138] Key type lgssc registered [ 7749.773132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7758.847643] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:33:38 (1769762018) [ 7760.897772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7760.905156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.986288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7761.996473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7762.030913] Lustre: lustre-MDT0000: new disk, initializing [ 7762.051143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7762.056664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7763.302033] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7767.154417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7767.188584] Lustre: 285786:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7767.208922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7767.210858] Lustre: Skipped 1 previous similar message [ 7767.242859] Lustre: lustre-MDT0001: new disk, initializing [ 7767.270229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7767.282746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7767.285242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7768.454349] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7771.311121] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7772.344101] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7774.311514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7774.402581] Lustre: lustre-OST0000: new disk, initializing [ 7774.404603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7774.429115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7775.092489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7775.096983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7775.128082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7776.103019] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7778.955719] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7784.749087] LustreError: 288221:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7784.783393] Lustre: server umount lustre-OST0000 complete [ 7788.000876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7788.003955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7788.007641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7790.561429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7790.561864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.683358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7795.685818] Lustre: Skipped 2 previous similar messages [ 7795.687444] LustreError: 288422:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7795.689590] LustreError: 288422:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7795.720115] Lustre: server umount lustre-MDT0000 complete [ 7796.971543] LustreError: 285776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762056 with bad export cookie 17976289500516329617 [ 7796.974031] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7796.974741] LustreError: 285776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7806.436912] LustreError: 288624:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7806.438844] LustreError: 288624:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7806.484738] Lustre: server umount lustre-MDT0001 complete [ 7810.159105] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 7811.116723] Key type lgssc unregistered [ 7811.244372] LNet: 289156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7811.247099] LNetError: 289156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7811.253410] LNet: Removed LNI 192.168.206.146@tcp [ 7811.562479] Key type .llcrypt unregistered [ 7811.563378] Key type ._llcrypt unregistered [ 7820.943237] Key type ._llcrypt registered [ 7820.945363] Key type .llcrypt registered [ 7820.983452] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7821.397040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7821.409232] alg: No test for adler32 (adler32-zlib) [ 7822.263843] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7822.360670] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 7823.936175] Key type lgssc registered [ 7824.342852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7826.958439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7826.964347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.122392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7829.545244] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7832.396141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7832.518787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7833.828376] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7835.575424] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7836.722532] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7838.989670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7839.113819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7841.153576] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7843.041112] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7861.131832] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7861.668813] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7863.264805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7863.268464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.274635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7863.778170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.779312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7863.783391] Lustre: Skipped 1 previous similar message [ 7863.786969] Lustre: Skipped 2 previous similar messages [ 7865.147824] LustreError: 293030:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7865.179307] Lustre: server umount lustre-MDT0000 complete [ 7866.383085] LustreError: 290509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762126 with bad export cookie 7330401223530137133 [ 7866.386748] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7866.387901] LustreError: 290509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7868.692049] LustreError: 290525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7868.697017] LustreError: 290525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7868.698077] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.46@tcp (stopping) [ 7868.897777] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.904923] LustreError: 290526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7871.905288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7871.909618] LustreError: 290526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7871.911396] Lustre: Skipped 1 previous similar message [ 7872.510454] LustreError: 293233:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7872.512299] LustreError: 293233:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7872.560519] Lustre: server umount lustre-MDT0001 complete [ 7878.937799] Lustre: lustre-OST0000: Not available for connect from 192.168.206.46@tcp (stopping) [ 7888.352151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7888.371103] LustreError: 293436:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7888.372774] LustreError: 293436:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7888.416312] Lustre: server umount lustre-OST0000 complete [ 7891.331080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7892.685801] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7893.605815] Lustre: 294281:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.146@tcp [ 7894.262759] LustreError: 294378:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7894.264745] LustreError: 294378:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7894.289911] Lustre: server umount MGS complete [ 7897.012710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.155852] LustreError: 294852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.177053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.409519] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7901.210162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7901.336941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7902.731669] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7905.703457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7907.710940] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7909.553889] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7915.232424] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 7916.423710] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7920.046699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 7922.146664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7922.148158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7922.149332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7922.151748] Lustre: Skipped 1 previous similar message [ 7922.156678] Lustre: Skipped 3 previous similar messages [ 7927.213072] LustreError: 297092:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7927.215412] LustreError: 297092:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7927.250410] Lustre: server umount lustre-OST0000 complete [ 7932.385937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7932.385955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7934.612664] Lustre: server umount lustre-MDT0000 complete [ 7935.842379] LustreError: 294832:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762195 with bad export cookie 7330401223530137959 [ 7935.845421] LustreError: 294832:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7935.846378] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7935.911173] LustreError: 297495:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7935.914441] LustreError: 297495:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7935.965870] Lustre: server umount lustre-MDT0001 complete [ 7939.689602] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 7940.736451] Key type lgssc unregistered [ 7940.876378] LNet: 298026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7940.879758] LNetError: 298026:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7940.890400] LNet: Removed LNI 192.168.206.146@tcp [ 7941.246648] Key type .llcrypt unregistered [ 7941.249221] Key type ._llcrypt unregistered [ 7947.181952] Key type ._llcrypt registered [ 7947.183236] Key type .llcrypt registered [ 7947.218944] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 7953.359908] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7953.681562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7953.715367] alg: No test for adler32 (adler32-zlib) [ 7954.581790] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7954.665479] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 7956.248106] Key type lgssc registered [ 7956.623986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7968.065703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7968.070961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7969.157511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7969.169418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7969.207277] Lustre: lustre-MDT0000: new disk, initializing [ 7969.231101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7969.238029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7970.493146] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7974.463083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7974.501702] Lustre: 301119:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7974.525428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7974.527575] Lustre: Skipped 1 previous similar message [ 7974.563608] Lustre: lustre-MDT0001: new disk, initializing [ 7974.596351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7974.610334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7974.613808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7975.868891] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7978.913479] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7980.106713] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7982.313490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7982.453070] Lustre: lustre-OST0000: new disk, initializing [ 7982.455777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7982.490702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7984.405750] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7987.486980] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7988.872322] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7989.753393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7989.756412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7989.787953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7989.969107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7991.372277] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7991.449494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7994.851499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7994.851535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.852964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7994.853775] LustreError: Skipped 1 previous similar message [ 7994.859211] Lustre: Skipped 1 previous similar message [ 7998.191091] LustreError: 304055:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7998.230222] Lustre: server umount lustre-OST0000 complete [ 7999.969548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7999.970270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7999.974794] Lustre: Skipped 1 previous similar message [ 7999.976545] Lustre: Skipped 1 previous similar message [ 8005.089237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8005.092052] Lustre: Skipped 2 previous similar messages [ 8005.633692] LustreError: 304256:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8005.635807] LustreError: 304256:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8005.673146] Lustre: server umount lustre-MDT0000 complete [ 8006.935312] LustreError: 301110:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762266 with bad export cookie 2142159117046747230 [ 8006.938489] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8006.940270] LustreError: 301110:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8007.026514] LustreError: 304458:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8007.028286] LustreError: 304458:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8007.080246] Lustre: server umount lustre-MDT0001 complete [ 8009.959173] Lustre: DEBUG MARKER: == conf-sanity test 200a: check CPU partitions =========== 03:37:49 (1769762269) [ 8027.203510] Lustre: DEBUG MARKER: == conf-sanity test 200b: set CPU pattern using core selection ========================================================== 03:38:06 (1769762286) [ 8042.371226] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8043.465038] Key type lgssc unregistered [ 8043.605386] LNet: 307664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8043.608087] LNetError: 307664:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8043.616398] LNet: Removed LNI 192.168.206.146@tcp [ 8043.912346] Key type .llcrypt unregistered [ 8043.913382] Key type ._llcrypt unregistered [ 8071.135503] Key type ._llcrypt registered [ 8071.136728] Key type .llcrypt registered [ 8071.180449] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8071.286262] Key type .llcrypt unregistered [ 8071.287187] Key type ._llcrypt unregistered [ 8082.484346] Key type ._llcrypt registered [ 8082.485852] Key type .llcrypt registered [ 8082.530329] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8082.635452] Key type .llcrypt unregistered [ 8082.636472] Key type ._llcrypt unregistered [ 8093.995138] Key type ._llcrypt registered [ 8093.996103] Key type .llcrypt registered [ 8094.035922] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8094.146236] Key type .llcrypt unregistered [ 8094.147162] Key type ._llcrypt unregistered [ 8121.955945] Key type ._llcrypt registered [ 8121.956761] Key type .llcrypt registered [ 8121.994859] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8122.111603] Key type .llcrypt unregistered [ 8122.113350] Key type ._llcrypt unregistered [ 8132.948455] Key type ._llcrypt registered [ 8132.949516] Key type .llcrypt registered [ 8132.985256] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8133.076174] Key type .llcrypt unregistered [ 8133.077159] Key type ._llcrypt unregistered [ 8143.662324] Key type ._llcrypt registered [ 8143.663405] Key type .llcrypt registered [ 8143.697740] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8143.781296] Key type .llcrypt unregistered [ 8143.782275] Key type ._llcrypt unregistered [ 8154.649981] Key type ._llcrypt registered [ 8154.651474] Key type .llcrypt registered [ 8154.690697] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8154.806285] Key type .llcrypt unregistered [ 8154.807406] Key type ._llcrypt unregistered [ 8165.330138] Key type ._llcrypt registered [ 8165.330947] Key type .llcrypt registered [ 8165.363788] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8165.448254] Key type .llcrypt unregistered [ 8165.449147] Key type ._llcrypt unregistered [ 8176.130383] Key type ._llcrypt registered [ 8176.131307] Key type .llcrypt registered [ 8176.171898] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8176.282227] Key type .llcrypt unregistered [ 8176.283850] Key type ._llcrypt unregistered [ 8197.596726] Key type ._llcrypt registered [ 8197.597767] Key type .llcrypt registered [ 8197.637278] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8197.729449] Key type .llcrypt unregistered [ 8197.730551] Key type ._llcrypt unregistered [ 8208.848933] Key type ._llcrypt registered [ 8208.850092] Key type .llcrypt registered [ 8208.889715] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8208.987272] Key type .llcrypt unregistered [ 8208.988590] Key type ._llcrypt unregistered [ 8220.237738] Key type ._llcrypt registered [ 8220.239280] Key type .llcrypt registered [ 8220.279678] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8220.384320] Key type .llcrypt unregistered [ 8220.386111] Key type ._llcrypt unregistered [ 8239.215180] Key type ._llcrypt registered [ 8239.216534] Key type .llcrypt registered [ 8239.254920] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 8245.967086] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 8246.271079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8246.279139] alg: No test for adler32 (adler32-zlib) [ 8247.128097] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 8247.205067] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 8248.784149] Key type lgssc registered [ 8249.140996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8259.253041] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7938 sec ========== 03:41:58 (1769762518) [ 8259.789088] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:41:59 (1769762519) === [ 8260.881856] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:42:00 (1769762520) === [ 8273.513581] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8274.521798] Key type lgssc unregistered [ 8274.647392] LNet: 326426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8274.649498] LNetError: 326426:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8274.659308] LNet: Removed LNI 192.168.206.146@tcp [ 8274.951347] Key type .llcrypt unregistered [ 8274.952309] Key type ._llcrypt unregistered