[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477585390 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002323] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009137] LSM: Security Framework initializing [ 0.010052] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012094] *** VALIDATE selinux *** [ 0.020508] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025195] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028122] *** VALIDATE tmpfs *** [ 0.029435] *** VALIDATE proc *** [ 0.031189] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033272] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036032] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.039628] debug: unmapping init [mem 0xffffffff84059000-0xffffffff84060fff] [ 0.041184] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042706] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043023] ... version: 2 [ 0.044014] ... bit width: 48 [ 0.045016] ... generic registers: 4 [ 0.046015] ... value mask: 0000ffffffffffff [ 0.047016] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049014] ... event mask: 000000070000000f [ 0.051225] rcu: Hierarchical SRCU implementation. [ 0.053455] smp: Bringing up secondary CPUs ... [ 0.054598] x86: Booting SMP configuration: [ 0.055027] .... node #0, CPUs: #1 #2 #3 [ 0.058716] smp: Brought up 1 node, 4 CPUs [ 0.060023] smpboot: Max logical packages: 1 [ 0.061014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.229417] node 0 deferred pages initialised in 166ms [ 0.233224] devtmpfs: initialized [ 0.234340] x86/mm: Memory block size: 128MB [ 0.237000] gcov: version magic: 0x41383552 [ 0.241347] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.248529] pinctrl core: initialized pinctrl subsystem [ 0.250254] [ 0.250724] ************************************************************* [ 0.254018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.256015] ** ** [ 0.258018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.261018] ** ** [ 0.264020] ** This means that this kernel is built to expose internal ** [ 0.266016] ** IOMMU data structures, which may compromise security on ** [ 0.268016] ** your system. ** [ 0.270015] ** ** [ 0.273020] ** If you see this message and you are not debugging the ** [ 0.276018] ** kernel, report this immediately to your vendor! ** [ 0.278017] ** ** [ 0.281016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.283015] ************************************************************* [ 0.286888] NET: Registered protocol family 16 [ 0.288550] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.292083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.295087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.299055] cpuidle: using governor menu [ 0.301042] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.304997] PCI: Using configuration type 1 for base access [ 0.308229] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.322153] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.323031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.325174] cryptd: max_cpu_qlen set to 1000 [ 0.328052] ACPI: Added _OSI(Module Device) [ 0.331058] ACPI: Added _OSI(Processor Device) [ 0.332024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.334022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.339407] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.347944] ACPI: Interpreter enabled [ 0.351104] ACPI: PM: (supports S0 S3 S4 S5) [ 0.353019] ACPI: Using IOAPIC for interrupt routing [ 0.355156] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.358478] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.372915] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.374063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.377030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.381095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.387700] acpiphp: Slot [2] registered [ 0.389170] acpiphp: Slot [5] registered [ 0.391194] acpiphp: Slot [6] registered [ 0.393190] acpiphp: Slot [7] registered [ 0.395222] acpiphp: Slot [8] registered [ 0.397173] acpiphp: Slot [9] registered [ 0.398176] acpiphp: Slot [10] registered [ 0.399155] acpiphp: Slot [3] registered [ 0.401125] acpiphp: Slot [4] registered [ 0.402105] acpiphp: Slot [11] registered [ 0.404110] acpiphp: Slot [12] registered [ 0.405117] acpiphp: Slot [13] registered [ 0.407157] acpiphp: Slot [14] registered [ 0.408152] acpiphp: Slot [15] registered [ 0.410144] acpiphp: Slot [16] registered [ 0.411157] acpiphp: Slot [17] registered [ 0.413186] acpiphp: Slot [18] registered [ 0.415188] acpiphp: Slot [19] registered [ 0.417163] acpiphp: Slot [20] registered [ 0.419149] acpiphp: Slot [21] registered [ 0.420118] acpiphp: Slot [22] registered [ 0.421133] acpiphp: Slot [23] registered [ 0.423156] acpiphp: Slot [24] registered [ 0.424136] acpiphp: Slot [25] registered [ 0.425163] acpiphp: Slot [26] registered [ 0.427153] acpiphp: Slot [27] registered [ 0.429155] acpiphp: Slot [28] registered [ 0.430131] acpiphp: Slot [29] registered [ 0.432159] acpiphp: Slot [30] registered [ 0.433126] acpiphp: Slot [31] registered [ 0.435095] PCI host bridge to bus 0000:00 [ 0.437042] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.439032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.442028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.444027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.447030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.450031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.452197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.455051] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.459448] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.469730] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.474049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.477026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.479021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.482025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.485779] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.488943] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.492055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.495000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.501027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.516030] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.522966] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.528838] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.541019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.552022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.567020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.577509] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.585022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.592026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.616027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.629205] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.635017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.642017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.666020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.676027] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.683023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.693023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.712022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.724101] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.734019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.744018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.761015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.771152] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.781028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.797025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.817021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.828326] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.830344] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.833457] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.836448] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.838287] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.844142] iommu: Default domain type: Passthrough [ 0.846451] SCSI subsystem initialized [ 0.847227] ACPI: bus type USB registered [ 0.849157] usbcore: registered new interface driver usbfs [ 0.851113] usbcore: registered new interface driver hub [ 0.853126] usbcore: registered new device driver usb [ 0.855290] pps_core: LinuxPPS API ver. 1 registered [ 0.856000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.859095] PTP clock support registered [ 0.861201] EDAC MC: Ver: 3.0.0 [ 0.863127] PCI: Using ACPI for IRQ routing [ 0.864941] NetLabel: Initializing [ 0.866013] NetLabel: domain hash size = 128 [ 0.867411] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.869094] NetLabel: unlabeled traffic allowed by default [ 0.872152] vgaarb: loaded [ 0.873544] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.875015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.881233] clocksource: Switched to clocksource kvm-clock [ 0.991567] VFS: Disk quotas dquot_6.6.0 [ 0.994837] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.997588] *** VALIDATE ramfs *** [ 0.998922] *** VALIDATE hugetlbfs *** [ 1.000295] pnp: PnP ACPI init [ 1.002826] pnp: PnP ACPI: found 6 devices [ 1.019339] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.022920] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.025219] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.027491] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.030278] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.032609] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.036266] NET: Registered protocol family 2 [ 1.038726] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.045224] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.048982] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.053898] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.056839] TCP: Hash tables configured (established 65536 bind 65536) [ 1.059709] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.062932] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.065800] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.069083] NET: Registered protocol family 1 [ 1.072408] RPC: Registered named UNIX socket transport module. [ 1.075029] RPC: Registered udp transport module. [ 1.076870] RPC: Registered tcp transport module. [ 1.078838] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.081401] NET: Registered protocol family 44 [ 1.083510] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.085548] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.087579] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.089756] PCI: CLS 0 bytes, default 64 [ 1.091310] Unpacking initramfs... [ 2.530345] debug: unmapping init [mem 0xffff8f95bcc54000-0xffff8f95bffbffff] [ 2.534256] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.536391] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.539326] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.050697] Initialise system trusted keyrings [ 3.053149] Key type blacklist registered [ 3.055872] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.068925] zbud: loaded [ 3.073369] *** VALIDATE nfs *** [ 3.075348] *** VALIDATE nfs4 *** [ 3.077633] pstore: using deflate compression [ 3.081867] Platform Keyring initialized [ 3.195454] NET: Registered protocol family 38 [ 3.197342] Key type asymmetric registered [ 3.199021] Asymmetric key parser 'x509' registered [ 3.201117] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.204186] io scheduler mq-deadline registered [ 3.206165] io scheduler kyber registered [ 3.208325] io scheduler bfq registered [ 3.210496] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.214148] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.217705] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.222370] ACPI: Power Button [PWRF] [ 3.228380] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.236145] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.253230] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.262394] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.276940] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.304030] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.331788] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.337945] Non-volatile memory driver v1.3 [ 3.339743] Linux agpgart interface v0.103 [ 3.373648] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.377365] vda: detected capacity change from 0 to 76689408 [ 3.394636] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.397579] vdb: detected capacity change from 0 to 1073741824 [ 3.413148] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.416814] vdc: detected capacity change from 0 to 2621440000 [ 3.431046] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.434128] vdd: detected capacity change from 0 to 2621440000 [ 3.457418] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.460614] vde: detected capacity change from 0 to 4294967296 [ 3.476593] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.480025] vdf: detected capacity change from 0 to 4294967296 [ 3.487638] libphy: Fixed MDIO Bus: probed [ 3.497525] usbcore: registered new interface driver usbserial_generic [ 3.499880] usbserial: USB Serial support registered for generic [ 3.502262] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.507106] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.508791] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.511224] mousedev: PS/2 mouse device common for all mice [ 3.513852] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.516380] rtc_cmos 00:05: RTC can wake from S4 [ 3.520206] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.520572] rtc_cmos 00:05: registered as rtc0 [ 3.524344] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.526911] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.527286] intel_pstate: CPU model not supported [ 3.532367] hid: raw HID events driver (C) Jiri Kosina [ 3.534385] usbcore: registered new interface driver usbhid [ 3.536849] usbhid: USB HID core driver [ 3.538834] drop_monitor: Initializing network drop monitor service [ 3.541530] Initializing XFRM netlink socket [ 3.543504] NET: Registered protocol family 10 [ 3.546973] Segment Routing with IPv6 [ 3.548435] NET: Registered protocol family 17 [ 3.550793] mpls_gso: MPLS GSO support [ 3.557051] RAS: Correctable Errors collector initialized. [ 3.559353] AVX version of gcm_enc/dec engaged. [ 3.561092] AES CTR mode by8 optimization enabled [ 3.637987] sched_clock: Marking stable (3637967269, 0)->(4576572244, -938604975) [ 3.641423] registered taskstats version 1 [ 3.643984] Loading compiled-in X.509 certificates [ 3.645439] zswap: loaded using pool lzo/zbud [ 3.680159] Key type big_key registered [ 3.691641] Key type encrypted registered [ 3.693265] ima: No TPM chip found, activating TPM-bypass! [ 3.695148] ima: Allocated hash algorithm: sha1 [ 3.696475] ima: No architecture policies found [ 3.697889] evm: Initialising EVM extended attributes: [ 3.699734] evm: security.selinux [ 3.700804] evm: security.ima [ 3.701841] evm: security.capability [ 3.702799] evm: HMAC attrs: 0x1 [ 3.704908] rtc_cmos 00:05: setting system clock to 2026-09-06 07:01:18 UTC (1788678078) [ 3.710698] debug: unmapping init [mem 0xffffffff85003000-0xffffffff851fffff] [ 3.713595] debug: unmapping init [mem 0xffffffff83d82000-0xffffffff84058fff] [ 3.722099] Write protecting the kernel read-only data: 28672k [ 3.726568] debug: unmapping init [mem 0xffffffff82403000-0xffffffff825fffff] [ 3.729607] debug: unmapping init [mem 0xffffffff82d14000-0xffffffff82dfffff] [ 3.764093] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.770765] systemd[1]: Detected virtualization kvm. [ 3.772581] systemd[1]: Detected architecture x86-64. [ 3.774095] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.801374] systemd[1]: No hostname configured. [ 3.803441] systemd[1]: Set hostname to . [ 3.807144] random: systemd: uninitialized urandom read (16 bytes read) [ 3.810163] systemd[1]: Initializing machine ID from random generator. [ 3.937557] random: systemd: uninitialized urandom read (16 bytes read) [ 3.939770] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.943443] random: systemd: uninitialized urandom read (16 bytes read) [ 3.946131] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.953743] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.577141] device-mapper: uevent: version 1.0.3 [ 4.578909] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.192359] virtio_net virtio0 ens2: renamed from eth0 [ 5.214969] random: fast init done [ 5.229247] scsi host0: ata_piix [ 5.256156] scsi host1: ata_piix [ 5.258360] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.260474] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.487938] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.206953] random: crng init done [ 10.208345] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.531611] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.645166] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.900879] SELinux: Disabled at runtime. [ 11.963987] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.973230] systemd[1]: Detected virtualization kvm. [ 11.975279] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.472330] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.475740] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.480522] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.484407] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.488193] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.494560] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.502513] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [FAILED] Failed to [ 12.578780] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.857945] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.085452] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.142658] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.266443] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.279537] EDAC sbridge: Ver: 1.1.2 [ 15.196296] Key type dns_resolver registered [ 15.498930] NFS: Registering the id_resolver key type [ 15.501159] Key type id_resolver registered [ 15.502404] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg630-server login: [ 56.296807] libcfs: loading out-of-tree module taints kernel. [ 56.357734] Key type ._llcrypt registered [ 56.360751] Key type .llcrypt registered [ 56.428472] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 73.704920] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 75.089320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.100520] alg: No test for adler32 (adler32-zlib) [ 76.600877] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 77.412043] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 79.175153] Key type lgssc registered [ 80.711308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.071982] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 134.745849] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 146.192112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 146.219675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.385971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.419414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.495851] Lustre: lustre-MDT0000: new disk, initializing [ 147.551673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.564978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 150.677485] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.600387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.685890] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 160.726391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 160.729800] Lustre: Skipped 1 previous similar message [ 160.818523] Lustre: lustre-MDT0001: new disk, initializing [ 160.896060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 160.916923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 160.924136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 164.058489] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.440598] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 176.030760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.282794] Lustre: lustre-OST0000: new disk, initializing [ 176.286229] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 176.291670] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 176.348146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.372441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.379421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.445657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 181.032870] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.335039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.447582] Lustre: lustre-OST0001: new disk, initializing [ 191.452633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 191.459643] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.523275] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.694659] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.126348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 197.141290] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 197.185543] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 207.835137] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 212.669178] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 219.547949] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing check_logdir /tmp/testlogs/ [ 224.593404] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing yml_node [ 228.961495] hrtimer: interrupt took 4938208 ns [ 228.984206] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 231.470667] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 234.078106] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 235.360787] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 6 03:05:08 EDT 2026 [ 250.573690] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 252.240841] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 268.768749] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.776047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 268.791118] Lustre: Skipped 3 previous similar messages [ 268.805467] Lustre: Skipped 3 previous similar messages [ 273.890921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.897986] Lustre: Skipped 3 previous similar messages [ 274.915728] Lustre: server umount lustre-MDT0000 complete [ 278.296789] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788678353 with bad export cookie 2057917572111993094 [ 278.298369] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 278.308647] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 279.008645] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.029337] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.043074] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.052754] Lustre: Skipped 2 previous similar messages [ 284.712103] Lustre: server umount lustre-MDT0001 complete [ 288.768486] Lustre: server umount lustre-OST0000 complete [ 292.501926] Lustre: server umount lustre-OST0001 complete [ 298.668863] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 307.018439] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 338.810841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 339.080872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 339.120593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 339.213561] Lustre: lustre-MDT0000: new disk, initializing [ 339.312661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 339.325381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 343.569384] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 352.536403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.633896] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 352.676229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 352.681269] Lustre: Skipped 1 previous similar message [ 352.744404] Lustre: lustre-MDT0001: new disk, initializing [ 352.856918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 352.899475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 352.912115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 357.208916] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 366.210198] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 371.415418] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 376.788299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 377.088359] Lustre: lustre-OST0000: new disk, initializing [ 377.091255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 377.096242] Lustre: 17037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 377.163318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 378.788572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 378.798399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 378.904850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 382.118165] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 390.672312] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 394.903229] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 395.095514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 399.118229] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 399.334796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 404.450106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 404.456353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.461771] LustreError: Skipped 1 previous similar message [ 404.462581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 404.490099] Lustre: Skipped 2 previous similar messages [ 407.363453] Lustre: server umount lustre-OST0000 complete [ 414.176994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 414.183875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 414.196590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 414.202330] Lustre: Skipped 1 previous similar message [ 418.811814] Lustre: server umount lustre-MDT0000 complete [ 419.811618] LustreError: 15079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 419.830617] LustreError: 15079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 421.997401] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788678496 with bad export cookie 2057917572111994627 [ 422.005501] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 422.010349] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 422.348344] Lustre: server umount lustre-MDT0001 complete [ 428.696853] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 03:08:21 (1788678501) [ 445.589658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.010585] LustreError: 19910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.091675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.202766] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 451.566781] LustreError: 19911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.654087] LustreError: 19910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.799886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.065552] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 468.092715] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 473.247335] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 475.043169] Lustre: Failing over lustre-MDT0000 [ 475.059227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.075545] Lustre: Skipped 1 previous similar message [ 475.081707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 475.088040] Lustre: Skipped 2 previous similar messages [ 475.267983] Lustre: server umount lustre-MDT0000 complete [ 480.224810] LustreError: 20638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.234848] LustreError: 20638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 492.513250] LustreError: 19907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.523960] LustreError: 19907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 493.704959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.827518] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.993900] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 499.169677] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 499.174720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 499.195624] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 506.400821] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 513.511088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.514299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 513.523341] Lustre: Skipped 2 previous similar messages [ 513.530314] Lustre: Skipped 1 previous similar message [ 516.369682] Lustre: server umount lustre-MDT0000 complete [ 518.626858] LustreError: 19910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.660401] LustreError: 19910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 520.001644] LustreError: 19891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788678594 with bad export cookie 2057917572111996041 [ 520.011335] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.016407] LustreError: 19891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 520.415791] Lustre: server umount lustre-MDT0001 complete [ 528.851473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.368653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.376279] Lustre: Skipped 2 previous similar messages [ 533.318581] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 541.933737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.394402] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 554.320095] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 558.929497] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 561.003762] Lustre: Failing over lustre-MDT0000 [ 561.194937] Lustre: server umount lustre-MDT0000 complete [ 563.169286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 563.178707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.212475] LustreError: 23246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.241730] LustreError: 23246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 580.105080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.239855] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 584.547453] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 585.704439] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 585.708882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 585.729748] Lustre: Skipped 1 previous similar message [ 585.753414] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 592.238168] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 595.936857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.936895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 595.937873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 595.951990] Lustre: Skipped 2 previous similar messages [ 601.538802] Lustre: server umount lustre-MDT0000 complete [ 604.598099] LustreError: 23227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788678679 with bad export cookie 2057917572111996979 [ 604.613277] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 604.618652] LustreError: 23227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 604.943840] Lustre: server umount lustre-MDT0001 complete [ 612.509881] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 03:11:25 (1788678685) [ 627.945502] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 637.067472] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 678.264807] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 689.063832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.331217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 689.367605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 689.471308] Lustre: lustre-MDT0000: new disk, initializing [ 689.533333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 689.539734] Lustre: Skipped 2 previous similar messages [ 689.558615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 692.909809] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 702.139326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.230668] Lustre: 29617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 702.441372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 706.071647] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 714.492875] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 719.997637] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 726.778949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 727.044272] Lustre: 31587:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 728.277277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 728.323487] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 728.407315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 728.430825] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 733.708922] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 742.460672] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 747.613816] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 769.504193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 769.521990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.554583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 769.564519] Lustre: Skipped 3 previous similar messages [ 775.552459] Lustre: server umount lustre-OST0000 complete [ 784.360035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 784.364646] LustreError: Skipped 1 previous similar message [ 785.664718] Lustre: server umount lustre-MDT0000 complete [ 789.214721] LustreError: 29608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788678864 with bad export cookie 2057917572111997385 [ 789.226766] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 789.228700] LustreError: 29608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 789.520879] Lustre: server umount lustre-MDT0001 complete [ 800.083861] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 802.896339] Key type lgssc unregistered [ 803.304656] LNet: 33559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.316080] LNetError: 33559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.328663] LNet: Removed LNI 192.168.206.130@tcp [ 804.780157] Key type .llcrypt unregistered [ 804.782289] Key type ._llcrypt unregistered [ 826.778973] Key type ._llcrypt registered [ 826.781549] Key type .llcrypt registered [ 826.909676] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 827.825192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.836781] alg: No test for adler32 (adler32-zlib) [ 828.827708] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 828.985410] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 830.615765] Key type lgssc registered [ 831.619251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 839.714739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 839.758225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.249820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.121952] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 850.410388] LustreError: 34970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.421607] LustreError: 34970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 853.970282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.204499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 857.496418] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 864.294810] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 869.631776] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 875.419519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 875.798023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 878.890179] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 880.751182] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 886.100513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 887.522856] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 900.689394] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 03:16:13 (1788678973) [ 902.241830] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 903.425753] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 908.219420] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 03:16:21 (1788678981) [ 915.256353] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 03:16:28 (1788678988) [ 915.932389] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 922.258542] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 922.263543] Lustre: Skipped 2 previous similar messages [ 923.652465] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 03:16:36 (1788678996) [ 926.514253] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 926.520704] Lustre: Skipped 3 previous similar messages [ 941.178919] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 941.185856] Lustre: Skipped 6 previous similar messages [ 943.939937] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 03:16:56 (1788679016) [ 947.239346] Lustre: *** cfs_fail_loc=131b, val=2*** [ 951.576236] Lustre: *** cfs_fail_loc=131b, val=2*** [ 962.995185] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 03:17:15 (1788679035) [ 965.114936] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 965.119674] Lustre: Skipped 3 previous similar messages [ 977.892452] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 03:17:31 (1788679051) [ 993.766326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.770356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 993.771793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 993.797739] Lustre: Skipped 1 previous similar message [ 998.781545] Lustre: server umount lustre-OST0000 complete [ 1002.976575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1002.983290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.999934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1003.010908] Lustre: Skipped 1 previous similar message [ 1004.001279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.002685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1008.511145] Lustre: server umount lustre-MDT0000 complete [ 1009.121710] LustreError: 34969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.132711] LustreError: 36813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679086 with bad export cookie 8735788480220111651 [ 1012.142490] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.146905] LustreError: 36813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1012.467148] Lustre: server umount lustre-MDT0001 complete [ 1024.607181] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 1027.429219] Key type lgssc unregistered [ 1027.743730] LNet: 42812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.751896] LNetError: 42812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.772089] LNet: Removed LNI 192.168.206.130@tcp [ 1028.677196] Key type .llcrypt unregistered [ 1028.680070] Key type ._llcrypt unregistered [ 1044.291759] Key type ._llcrypt registered [ 1044.294954] Key type .llcrypt registered [ 1044.405940] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 1059.082961] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 1059.828231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.863496] alg: No test for adler32 (adler32-zlib) [ 1060.926216] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 1061.130617] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 1062.767546] Key type lgssc registered [ 1063.805655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1095.368052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1095.406525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.742231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1096.787866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1096.875529] Lustre: lustre-MDT0000: new disk, initializing [ 1096.935361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1096.950212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1101.602266] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1110.828833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.968637] Lustre: 45964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1111.025184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1111.032711] Lustre: Skipped 1 previous similar message [ 1111.162643] Lustre: lustre-MDT0001: new disk, initializing [ 1111.255511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1111.300115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1111.313744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1115.046206] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1123.344871] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1128.260409] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1134.665948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1134.948494] Lustre: lustre-OST0000: new disk, initializing [ 1134.952273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1134.958823] Lustre: 47932:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1135.036817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.487391] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1143.856851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1143.874204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1143.950781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1148.682973] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1152.770610] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1152.958818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1156.978793] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1157.153107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1159.652076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.665343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1159.666848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1159.674601] Lustre: Skipped 1 previous similar message [ 1164.776219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1164.785149] Lustre: Skipped 2 previous similar messages [ 1165.109378] Lustre: server umount lustre-OST0000 complete [ 1169.894757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.897041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.912186] Lustre: Skipped 1 previous similar message [ 1169.921732] Lustre: Skipped 1 previous similar message [ 1176.846047] Lustre: server umount lustre-MDT0000 complete [ 1180.128557] LustreError: 45975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.160791] LustreError: 45975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1181.141195] LustreError: 45956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679255 with bad export cookie 16813718101917273980 [ 1181.144984] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1181.150219] LustreError: 45956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1181.428948] Lustre: server umount lustre-MDT0001 complete [ 1191.006855] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 03:21:03 (1788679263) [ 1197.326507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1197.699519] LustreError: 50391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.760802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1201.224271] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1203.169489] LustreError: 50392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.270134] LustreError: 50391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.044357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.402254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1213.375898] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1220.096808] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1224.718909] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1230.306047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1230.568711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.532794] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1242.678674] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1246.786384] LustreError: 52919:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1247.653489] Lustre: Modifying parameter general.timeout=1 in log params [ 1249.041483] Lustre: Modifying parameter general.timeout=3 in log params [ 1249.046837] Lustre: Skipped 1 previous similar message [ 1251.118289] Lustre: Modifying parameter general.timeout=6 in log params [ 1251.121764] Lustre: Skipped 2 previous similar messages [ 1255.438274] Lustre: Modifying parameter general.timeout=12 in log params [ 1255.441572] Lustre: Skipped 5 previous similar messages [ 1264.180596] Lustre: Modifying parameter general.timeout=23 in log params [ 1264.187194] Lustre: Skipped 10 previous similar messages [ 1280.913185] Lustre: Modifying parameter general.timeout=44 in log params [ 1280.917704] Lustre: Skipped 20 previous similar messages [ 1313.468882] Lustre: Modifying parameter general.timeout=89 in log params [ 1313.472658] Lustre: Skipped 44 previous similar messages [ 1353.356401] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 03:23:46 (1788679426) [ 1356.018856] Lustre: 59845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1357.512143] Lustre: 59943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1360.834706] Lustre: 60162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1360.842293] Lustre: 60162:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1378.979308] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1378.993188] Lustre: Skipped 59 previous similar messages [ 1386.652466] Lustre: 61713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1386.663600] Lustre: 61713:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1386.699944] LustreError: 61713:0:(mgs_handler.c:1205:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1390.361687] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 03:24:23 (1788679463) [ 1401.835646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.837095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.859665] Lustre: Skipped 2 previous similar messages [ 1401.867161] Lustre: Skipped 2 previous similar messages [ 1406.512500] Lustre: server umount lustre-MDT0000 complete [ 1406.946437] LustreError: 50388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1406.965023] LustreError: 50388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1409.521119] LustreError: 54057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679484 with bad export cookie 16813718101917274862 [ 1409.522972] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1409.544281] LustreError: 54057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1409.776822] Lustre: server umount lustre-MDT0001 complete [ 1423.645386] Lustre: server umount lustre-OST0000 complete [ 1450.393378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.660446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1450.868834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.234385] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1463.445379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.555159] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1463.576817] Lustre: 64244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1468.260687] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1476.330300] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1482.544178] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1488.943523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1489.087259] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1489.366334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.376644] Lustre: Skipped 1 previous similar message [ 1495.675099] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1502.738308] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1504.870336] Lustre: 66811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1510.880557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1510.888505] LustreError: Skipped 1 previous similar message [ 1510.893913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.907169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1516.775297] Lustre: server umount lustre-OST0000 complete [ 1521.632513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.643114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1521.660981] Lustre: Skipped 2 previous similar messages [ 1521.675291] Lustre: Skipped 5 previous similar messages [ 1526.826081] Lustre: server umount lustre-MDT0000 complete [ 1530.326237] LustreError: 64236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679605 with bad export cookie 16813718101917277683 [ 1530.332794] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.340665] LustreError: 64236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1530.647585] Lustre: server umount lustre-MDT0001 complete [ 1541.898318] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 1544.581217] Key type lgssc unregistered [ 1544.848737] LNet: 68017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.861670] LNetError: 68017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.884224] LNet: Removed LNI 192.168.206.130@tcp [ 1545.694268] Key type .llcrypt unregistered [ 1545.698089] Key type ._llcrypt unregistered [ 1574.829692] Key type ._llcrypt registered [ 1574.831984] Key type .llcrypt registered [ 1574.930539] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 1575.979875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1576.016423] alg: No test for adler32 (adler32-zlib) [ 1577.158769] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 1577.459965] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 1579.186311] Key type lgssc registered [ 1580.969832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1589.912650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1589.959899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.598129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.188250] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1602.027102] LustreError: 69426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1602.043863] LustreError: 69426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1606.237396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.875700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1611.717985] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1619.355690] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1624.748464] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1631.152148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1631.694449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.434815] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1643.731176] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1652.278385] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1662.442316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.449953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1662.470143] Lustre: Skipped 2 previous similar messages [ 1662.477897] Lustre: Skipped 2 previous similar messages [ 1667.555739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.562263] Lustre: Skipped 1 previous similar message [ 1668.403520] Lustre: server umount lustre-MDT0000 complete [ 1671.895317] LustreError: 72113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679746 with bad export cookie 3808122891476829143 [ 1671.909681] LustreError: 72113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1671.909831] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1672.184631] Lustre: server umount lustre-MDT0001 complete [ 1686.273067] Lustre: server umount lustre-OST0000 complete [ 1713.413545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.600518] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1713.630136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1713.795699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.996727] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1723.733870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.849496] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1723.873983] Lustre: 74306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1724.283106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1728.141483] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1734.260532] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1739.092874] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1744.625057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1744.763017] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1745.049703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1750.954439] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1758.887961] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1760.690767] Lustre: 76879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1762.414377] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1770.465885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.467415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1770.472906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1770.485576] Lustre: Skipped 1 previous similar message [ 1770.504547] LustreError: Skipped 1 previous similar message [ 1773.504662] Lustre: server umount lustre-OST0000 complete [ 1780.705478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.707421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1780.733532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.741528] Lustre: Skipped 2 previous similar messages [ 1783.393751] Lustre: server umount lustre-MDT0000 complete [ 1785.824330] LustreError: 75727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.844041] LustreError: 75727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1786.691491] LustreError: 74299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679861 with bad export cookie 3808122891476829990 [ 1786.694978] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1786.700408] LustreError: 74299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1787.007236] Lustre: server umount lustre-MDT0001 complete [ 1798.631813] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 1801.547739] Key type lgssc unregistered [ 1801.869898] LNet: 78088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.888967] LNetError: 78088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.906621] LNet: Removed LNI 192.168.206.130@tcp [ 1802.699463] Key type .llcrypt unregistered [ 1802.705173] Key type ._llcrypt unregistered [ 1829.379559] Key type ._llcrypt registered [ 1829.383508] Key type .llcrypt registered [ 1829.520841] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 1830.620141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1830.668057] alg: No test for adler32 (adler32-zlib) [ 1831.660746] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 1831.856860] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 1833.511973] Key type lgssc registered [ 1834.924079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1841.934257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1841.963557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.407404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.037436] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1853.922568] LustreError: 79509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1853.950644] LustreError: 79509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1855.815447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.352766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1859.707508] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1867.202471] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1874.530564] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1883.168173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1884.012110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.008283] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1898.393645] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1907.278963] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1908.147741] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1910.028280] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1910.042449] Lustre: Skipped 1 previous similar message [ 1912.693366] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1912.700522] Lustre: Skipped 2 previous similar messages [ 1917.349402] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1917.355325] Lustre: Skipped 4 previous similar messages [ 1926.107119] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1926.117822] Lustre: Skipped 9 previous similar messages [ 1942.325892] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1942.332222] Lustre: Skipped 18 previous similar messages [ 1975.145476] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1975.156068] Lustre: Skipped 38 previous similar messages [ 2000.294443] Lustre: 87604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2002.791186] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 03:34:35 (1788680075) [ 2004.300642] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2005.736163] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 03:34:38 (1788680078) [ 2012.170257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2012.185426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.203678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2013.158586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2013.162295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.722784] Lustre: server umount lustre-OST0000 complete [ 2020.320310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2020.326356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.340623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.347645] Lustre: Skipped 1 previous similar message [ 2023.393508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.396379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2023.410629] Lustre: Skipped 1 previous similar message [ 2025.283062] Lustre: server umount lustre-MDT0000 complete [ 2028.512017] LustreError: 79510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2028.537891] LustreError: 79510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2028.721710] LustreError: 79490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680103 with bad export cookie 13352723070673059450 [ 2028.727968] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.733886] LustreError: 79490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2029.189645] Lustre: server umount lustre-MDT0001 complete [ 2040.562384] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 2043.268992] Key type lgssc unregistered [ 2043.574955] LNet: 88851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.584523] LNetError: 88851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.599778] LNet: Removed LNI 192.168.206.130@tcp [ 2044.523301] Key type .llcrypt unregistered [ 2044.526805] Key type ._llcrypt unregistered [ 2049.243891] Key type ._llcrypt registered [ 2049.248437] Key type .llcrypt registered [ 2049.352403] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2054.209494] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules [ 2055.390780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2055.420610] alg: No test for adler32 (adler32-zlib) [ 2056.644513] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 2056.973457] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 2058.735355] Key type lgssc registered [ 2060.087178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2061.368599] LustreError: 90059:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2079.460509] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 2082.540115] LustreError: 90059:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2087.252053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2087.276334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.813224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.198627] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2099.175980] LustreError: 91075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.202947] LustreError: 91075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2103.411563] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 03:36:16 (1788680176) [ 2104.294921] LustreError: 91074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.043353] LustreError: 91106:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2109.053902] LustreError: 91106:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2109.059307] LustreError: 91106:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2109.516787] Lustre: server umount lustre-MDT0000 complete [ 2123.003915] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 2125.614760] Key type lgssc unregistered [ 2125.944541] LNet: 92600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.951063] LNetError: 92600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.963375] LNet: Removed LNI 192.168.206.130@tcp [ 2126.683163] Key type .llcrypt unregistered [ 2126.686370] Key type ._llcrypt unregistered [ 2143.784133] Key type ._llcrypt registered [ 2143.787650] Key type .llcrypt registered [ 2143.882379] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 2144.899506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2144.915656] alg: No test for adler32 (adler32-zlib) [ 2146.003969] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 2146.178303] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 2147.807187] Key type lgssc registered [ 2149.005514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2155.498838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2155.521341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.908806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.439165] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2166.244976] LustreError: 93818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.262618] LustreError: 93818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2167.593986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.211025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2172.163570] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2179.821404] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2185.392837] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2191.945612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2192.675781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2198.524161] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2206.108311] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2251.446354] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 03:38:44 (1788680324) [ 2253.533681] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2255.647807] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 03:38:48 (1788680328) [ 2259.940503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2259.949548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.971168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.520268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.521470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.537109] Lustre: Skipped 1 previous similar message [ 2265.857393] Lustre: server umount lustre-MDT0000 complete [ 2268.647930] LustreError: 93817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.668614] LustreError: 93817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2269.333130] LustreError: 93798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680344 with bad export cookie 17994254361378335797 [ 2269.337704] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2269.344327] LustreError: 93798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2269.692503] Lustre: server umount lustre-MDT0001 complete [ 2284.052368] Lustre: server umount lustre-OST0000 complete [ 2294.209238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.548899] LustreError: 98089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.706677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.664206] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2299.873366] LustreError: 98090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.995575] LustreError: 98089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.074263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.739351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2310.709685] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2318.058204] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2324.188967] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2337.622491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2337.812134] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2337.823490] LustreError: 98079:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2337.851574] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2337.859983] LustreError: 100172:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -98 [ 2337.868593] LustreError: 100172:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 2337.880542] LustreError: 100172:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2337.958826] Lustre: server umount lustre-OST0000 complete [ 2337.961933] LustreError: 100172:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2346.405613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2346.607983] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2346.623897] LustreError: 98078:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2346.655129] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2346.675976] LustreError: 100637:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -98 [ 2346.685910] LustreError: 100637:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 2346.692045] LustreError: 100637:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2346.787219] Lustre: server umount lustre-OST0000 complete [ 2346.793874] LustreError: 100637:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2356.592993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2356.796097] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2356.803984] Lustre: Found index 0 for lustre-OST0000, updating log [ 2356.811223] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2356.899876] Lustre: lustre-OST0000: new disk, initializing [ 2356.902904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2356.907769] Lustre: 101174:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2357.160150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.564882] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2367.256048] Lustre: Failing over lustre-OST0000 [ 2367.431520] Lustre: server umount lustre-OST0000 complete [ 2373.090190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2373.102115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.118499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2373.125991] Lustre: Skipped 2 previous similar messages [ 2376.672403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.681333] Lustre: Skipped 3 previous similar messages [ 2380.837561] Lustre: server umount lustre-MDT0000 complete [ 2381.798538] LustreError: 98090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.815787] LustreError: 98090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2384.308539] LustreError: 100654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680459 with bad export cookie 17994254361378338541 [ 2384.330483] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.330843] LustreError: 100654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2384.678156] Lustre: server umount lustre-MDT0001 complete [ 2393.622309] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 03:41:06 (1788680466) [ 2411.060314] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 2421.430809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.825176] LustreError: 103518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.966154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2426.577680] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2434.027178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.529320] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2441.203885] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2447.342293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2448.007435] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2454.079808] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2459.253375] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2459.259416] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2459.275765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2462.666557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2462.737411] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2462.794454] Lustre: lustre-OST0001: new disk, initializing [ 2462.796974] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2462.800573] Lustre: 105838:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2462.945581] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 2462.966541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2462.975853] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2462.982780] Lustre: Skipped 2 previous similar messages [ 2467.051127] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2467.088420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2467.111678] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2467.114602] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2467.123510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2469.519529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2469.543584] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2469.730033] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2470.194942] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2479.117539] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2490.883742] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2495.467816] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2495.473528] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2495.476360] Lustre: Skipped 2 previous similar messages [ 2501.466726] Lustre: server umount lustre-MDT0001 complete [ 2503.976601] LustreError: 103514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.007830] LustreError: 103514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2509.050161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.139975] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2509.145158] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2509.192706] Lustre: 103508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2510.319236] LustreError: 107383:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2510.327622] LustreError: 107383:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2510.338039] LustreError: 103510:0:(mgc_request.c:612:do_requeue()) failed processing log: -17 [ 2510.401568] LustreError: 107391:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.130@tcp: cfg command failed: rc = -17 [ 2510.410216] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2510.410216] [ 2513.243729] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2514.926183] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2521.259536] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 03:43:14 (1788680594) [ 2540.442418] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 2559.306653] Lustre: 109451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2580.760378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2584.623536] Lustre: 110586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2673.121317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2673.130850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.160432] Lustre: Skipped 1 previous similar message [ 2673.171727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2673.178790] Lustre: Skipped 6 previous similar messages [ 2677.391267] Lustre: server umount lustre-MDT0000 complete [ 2678.755459] LustreError: 103514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.773313] LustreError: 103514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2680.979102] LustreError: 106556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680755 with bad export cookie 17994254361378339542 [ 2680.980157] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2680.994960] LustreError: 106556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2681.254605] Lustre: server umount lustre-MDT0001 complete [ 2694.912665] Lustre: server umount lustre-OST0000 complete [ 2708.834520] Lustre: server umount lustre-OST0001 complete [ 2715.157512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2725.735661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2735.349722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2745.950631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2767.041482] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 2779.668541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2779.693725] Lustre: lustre-MDT0000: reset Object Index mappings [ 2780.859826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2780.866722] Lustre: Skipped 1 previous similar message [ 2786.331773] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2795.295203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.333113] Lustre: lustre-MDT0001: reset Object Index mappings [ 2801.143089] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2804.479443] Lustre: 115695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2811.363604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2817.996359] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2820.398311] LustreError: 116053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.410926] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2820.419921] LustreError: 116053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2825.561290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 2826.342582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2832.403551] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2832.409781] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2833.060240] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2840.905878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2844.804198] Lustre: 117576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2898.733801] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 03:49:31 (1788680971) [ 2924.515446] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2924.517741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2924.526630] Lustre: Skipped 5 previous similar messages [ 2924.535695] Lustre: Skipped 5 previous similar messages [ 2929.123476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2930.861055] Lustre: server umount lustre-MDT0000 complete [ 2934.344240] LustreError: 114525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788681009 with bad export cookie 17994254361378386659 [ 2934.347430] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2934.357493] LustreError: 114525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2940.879588] Lustre: server umount lustre-MDT0001 complete [ 2950.665223] Lustre: server umount lustre-OST0000 complete [ 2960.791975] Lustre: server umount lustre-OST0001 complete [ 2969.602889] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 2980.297446] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 3013.615560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.747449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3013.777124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3013.834691] Lustre: lustre-MDT0000: new disk, initializing [ 3013.877814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3013.881332] Lustre: Skipped 3 previous similar messages [ 3013.891637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3018.786524] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3025.267487] Lustre: server umount lustre-MDT0000 complete [ 3033.062079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3033.309440] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3033.319465] Lustre: 121843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3033.329966] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3036.417343] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3040.537074] Lustre: server umount lustre-MDT0000 complete [ 3049.252469] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 03:52:01 (1788681121) [ 3050.860369] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3052.823163] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 03:52:05 (1788681125) [ 3070.417418] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 3080.251809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.683424] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3093.942790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.032808] Lustre: 123894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3094.098581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3094.106381] Lustre: Skipped 1 previous similar message [ 3094.226458] Lustre: lustre-MDT0001: new disk, initializing [ 3094.366240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3094.387314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3098.782620] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3103.545776] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3110.209572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.485275] Lustre: lustre-OST0000: new disk, initializing [ 3110.489729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3110.499289] Lustre: 125470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3112.319508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3112.337653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3112.429355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3117.907418] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3129.085438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3129.218309] Lustre: lustre-OST0001: new disk, initializing [ 3129.221788] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3129.226625] Lustre: 126341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3130.400959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3130.419599] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3130.557015] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3136.690117] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3146.837626] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3150.689370] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3523.884778] Lustre: 132754:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3523.895209] Lustre: 132754:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3523.904121] Lustre: 132754:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3523.911473] Lustre: 132754:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3523.917649] Lustre: 132754:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3523.927185] Lustre: 132754:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3816.552994] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:04:48 (1788681888) [ 3831.778143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3831.781874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.792480] Lustre: Skipped 3 previous similar messages [ 3831.798881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3831.803737] Lustre: Skipped 8 previous similar messages [ 3834.048175] Lustre: server umount lustre-MDT0000 complete [ 3837.414947] LustreError: 127147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.430766] LustreError: 127147:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3837.475851] LustreError: 132798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788681912 with bad export cookie 17994254361378455665 [ 3837.494334] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3842.533420] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.541634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3842.545491] Lustre: Skipped 4 previous similar messages [ 3842.561483] Lustre: Skipped 5 previous similar messages [ 3844.026293] Lustre: server umount lustre-MDT0001 complete [ 3853.847261] Lustre: server umount lustre-OST0000 complete [ 3864.973361] Lustre: server umount lustre-OST0001 complete [ 3872.238642] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 3882.591573] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 3919.427988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3919.717978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3919.756970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3919.858179] Lustre: lustre-MDT0000: new disk, initializing [ 3919.942381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3919.947606] Lustre: Skipped 5 previous similar messages [ 3919.963110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3924.617500] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3935.259047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3935.400428] Lustre: 135822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3935.462590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3935.466342] Lustre: Skipped 1 previous similar message [ 3935.533752] Lustre: lustre-MDT0001: new disk, initializing [ 3935.656127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3935.672101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3940.495306] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3951.269390] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3956.702558] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3963.844277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3964.200433] Lustre: lustre-OST0000: new disk, initializing [ 3964.204114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3964.209993] Lustre: 137791:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3964.330447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.334714] Lustre: Skipped 1 previous similar message [ 3965.486236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3965.494452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3965.629484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3970.901582] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3979.973093] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3989.832769] Lustre: lustre-MDD0000: changelog on [ 3992.309918] Lustre: lustre-MDD0001: changelog on [ 5945.076106] Lustre: lustre-MDD0001: changelog off [ 5946.937875] Lustre: 136952:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5947.735847] Lustre: lustre-MDD0000: changelog off [ 5957.614092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5957.614219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5957.648029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5959.883494] Lustre: server umount lustre-OST0000 complete [ 5967.842635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.858907] Lustre: Skipped 1 previous similar message [ 5967.867725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5967.885849] Lustre: Skipped 2 previous similar messages [ 5970.335817] Lustre: server umount lustre-MDT0000 complete [ 5972.959983] LustreError: 136699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5972.992430] LustreError: 136699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5974.804840] LustreError: 135814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684049 with bad export cookie 17994254361379278214 [ 5974.814279] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5974.815171] LustreError: 135814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5975.208996] Lustre: server umount lustre-MDT0001 complete [ 5985.250665] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 5987.681962] Key type lgssc unregistered [ 5987.954193] LNet: 146949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5987.961631] LNetError: 146949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5987.976615] LNet: Removed LNI 192.168.206.130@tcp [ 5988.776209] Key type .llcrypt unregistered [ 5988.779231] Key type ._llcrypt unregistered [ 6007.547632] Key type ._llcrypt registered [ 6007.550696] Key type .llcrypt registered [ 6007.675409] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 6021.171341] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 6022.193465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6022.209130] alg: No test for adler32 (adler32-zlib) [ 6023.279954] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 6023.511742] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 6025.159172] Key type lgssc registered [ 6026.535265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6059.237554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6059.256442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.525536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6060.562988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6060.694489] Lustre: lustre-MDT0000: new disk, initializing [ 6060.806666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.825512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6065.633362] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6075.822782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6075.956742] Lustre: 150145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6076.020151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6076.024150] Lustre: Skipped 1 previous similar message [ 6076.086486] Lustre: lustre-MDT0001: new disk, initializing [ 6076.182371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6076.237574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6076.246143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6080.386669] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6088.963174] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6094.139700] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6101.519750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6101.765899] Lustre: lustre-OST0000: new disk, initializing [ 6101.770665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6101.780143] Lustre: 152115:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6101.863974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6109.238873] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6111.805688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6111.811071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6111.957253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6118.563783] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6124.105636] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_module obdecho/obdecho [ 6126.276827] LustreError: 153119:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6126.284834] LustreError: 153119:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6128.213231] LustreError: 153215:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6142.944676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6142.949423] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.952906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6142.966281] Lustre: Skipped 2 previous similar messages [ 6142.969617] Lustre: Skipped 1 previous similar message [ 6145.453853] Lustre: server umount lustre-MDT0000 complete [ 6149.244255] LustreError: 152114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684224 with bad export cookie 16841074925507212121 [ 6149.252974] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6149.255948] LustreError: 152114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6149.644667] Lustre: server umount lustre-MDT0001 complete [ 6163.757994] Lustre: server umount lustre-OST0000 complete [ 6186.720964] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 6196.359767] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 6234.004377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.251246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6234.298775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6234.357974] Lustre: lustre-MDT0000: new disk, initializing [ 6234.415131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6234.433724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6238.766435] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6249.494817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.604884] Lustre: 156930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6249.676751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6249.680807] Lustre: Skipped 1 previous similar message [ 6249.792794] Lustre: lustre-MDT0001: new disk, initializing [ 6249.800067] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6249.891042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6249.966056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6249.972037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6253.928135] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6262.403906] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6267.795788] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6274.515937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6274.799238] Lustre: lustre-OST0000: new disk, initializing [ 6274.802977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6274.810981] Lustre: 158896:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6274.921933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.220572] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6281.298771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6281.310991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6281.477057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6291.255582] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6297.205877] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6297.465728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6302.613115] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6302.808759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6307.302350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6307.310343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.311639] LustreError: Skipped 1 previous similar message [ 6307.334459] Lustre: Skipped 1 previous similar message [ 6307.338069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6307.350337] Lustre: Skipped 1 previous similar message [ 6312.427789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6312.447104] Lustre: Skipped 1 previous similar message [ 6317.541030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6317.555675] Lustre: Skipped 1 previous similar message [ 6319.071140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6319.293984] Lustre: server umount lustre-OST0000 complete [ 6326.752956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6326.768815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6326.783990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6329.186904] Lustre: server umount lustre-MDT0000 complete [ 6332.902298] LustreError: 156942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.923587] LustreError: 156942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6333.075107] LustreError: 156921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684407 with bad export cookie 16841074925507213080 [ 6333.079540] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.087591] LustreError: 156921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6333.398861] Lustre: server umount lustre-MDT0001 complete [ 6337.345454] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 04:46:49 (1788684409) [ 6339.129311] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6341.541317] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 04:46:54 (1788684414) [ 6350.420484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.945611] LustreError: 161313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6351.004502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.081567] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6356.451493] LustreError: 161314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6360.552289] LustreError: 161313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6365.403751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6370.434915] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6378.085750] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6383.575592] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6390.748889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6391.136957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6391.148551] Lustre: Skipped 1 previous similar message [ 6398.157902] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6406.530502] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6417.751688] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.30@tcp (stopping) [ 6417.758617] Lustre: Skipped 2 previous similar messages [ 6418.205376] Lustre: server umount lustre-MDT0000 complete [ 6420.965535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6420.971637] LustreError: 161310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.986923] Lustre: Skipped 2 previous similar messages [ 6421.011257] LustreError: 161310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6422.172386] LustreError: 162050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684496 with bad export cookie 16841074925507213955 [ 6422.177000] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6422.183374] LustreError: 162050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6426.091248] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.110281] Lustre: Skipped 1 previous similar message [ 6428.613874] Lustre: server umount lustre-MDT0001 complete [ 6435.294636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6449.129236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6449.394362] LustreError: 165156:0:(llog.c:1655:llog_backup()) MGC192.168.206.130@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6449.400634] Lustre: 165156:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.130@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6452.705547] LustreError: 165161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.730286] LustreError: 165161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6468.895880] LustreError: 165163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6468.922562] LustreError: 165163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6503.392729] LustreError: 165163:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6503.408570] LustreError: 165163:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 6510.047338] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6510.053178] Lustre: 165156:0:(mgc_request.c:1898:mgc_process_log()) MGC192.168.206.130@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6551.040940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6554.746657] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6556.137341] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6561.917359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.398517] LustreError: 165162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.401373] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe9b7846aba9893c0 [ 6567.413633] LustreError: 165162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 45 previous similar messages [ 6567.427997] Lustre: MGC192.168.206.130@tcp: Connection restored to 0@lo (at 0@lo) [ 6571.743659] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6573.039420] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6578.735705] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6580.194677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6585.087280] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6586.292732] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6603.748279] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6603.749971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6603.753640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6603.756261] Lustre: Skipped 2 previous similar messages [ 6603.771078] Lustre: Skipped 6 previous similar messages [ 6609.588754] Lustre: server umount lustre-MDT0000 complete [ 6614.006207] LustreError: 161293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684688 with bad export cookie 16841074925507220416 [ 6614.007674] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6614.015868] LustreError: 161293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6614.342763] Lustre: server umount lustre-MDT0001 complete [ 6628.978759] Lustre: server umount lustre-OST0000 complete [ 6650.521877] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 6659.813261] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 6691.848494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6692.112990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6692.161427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6692.264897] Lustre: lustre-MDT0000: new disk, initializing [ 6692.329454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6692.337763] Lustre: Skipped 1 previous similar message [ 6692.352687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6696.814372] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6706.867677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.955755] Lustre: 170262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6707.161814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6711.356530] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6721.384309] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6728.624689] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6737.511890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6737.911449] Lustre: lustre-OST0000: new disk, initializing [ 6737.918931] Lustre: Skipped 1 previous similar message [ 6737.926940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6737.933125] Lustre: Skipped 2 previous similar messages [ 6737.940597] Lustre: 172234:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6745.255466] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6747.734853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6747.745442] Lustre: Skipped 1 previous similar message [ 6747.748619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6747.841534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6755.144349] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6760.561459] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6760.812678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6765.765140] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6765.979766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6768.615892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6768.616099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6768.643234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6768.645366] LustreError: Skipped 1 previous similar message [ 6768.656618] Lustre: Skipped 4 previous similar messages [ 6774.297579] Lustre: server umount lustre-OST0000 complete [ 6778.873813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6784.638131] Lustre: server umount lustre-MDT0000 complete [ 6788.908535] LustreError: 170254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684863 with bad export cookie 16841074925507237244 [ 6788.924623] LustreError: 170254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6788.927909] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6789.381484] Lustre: server umount lustre-MDT0001 complete [ 6798.556314] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 04:54:31 (1788684871) [ 6805.443917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6805.957131] LustreError: 174644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6805.979956] LustreError: 174644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6811.898504] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6820.384897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6825.420979] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6832.436568] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6837.865770] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6844.063553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6849.753659] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6856.946285] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6866.154988] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 04:55:39 (1788684939) [ 6871.008855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6871.018416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6871.027270] Lustre: Skipped 3 previous similar messages [ 6875.792972] Lustre: server umount lustre-OST0000 complete [ 6886.247888] Lustre: server umount lustre-MDT0000 complete [ 6890.044795] LustreError: 175379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788684964 with bad export cookie 16841074925507238133 [ 6890.048239] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6890.058092] LustreError: 175379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6890.549627] Lustre: server umount lustre-MDT0001 complete [ 6901.259427] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 6903.755529] Key type lgssc unregistered [ 6904.036962] LNet: 178368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6904.042509] LNetError: 178368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6904.057451] LNet: Removed LNI 192.168.206.130@tcp [ 6904.903108] Key type .llcrypt unregistered [ 6904.904827] Key type ._llcrypt unregistered [ 6922.122540] Key type ._llcrypt registered [ 6922.129358] Key type .llcrypt registered [ 6922.238228] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 6923.648706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6923.682404] alg: No test for adler32 (adler32-zlib) [ 6924.713423] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 6924.918880] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 6926.633587] Key type lgssc registered [ 6927.699152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6934.996149] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6935.013232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6950.496636] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6955.615638] LustreError: 179609:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.130@tcp: failed processing log, type 4: rc = -110 [ 6960.735439] LustreError: 179609:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6960.752358] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6960.765397] LustreError: MGC192.168.206.130@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6960.772545] LustreError: 179609:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6960.779383] LustreError: 179609:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -22 [ 6960.785265] LustreError: 179609:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 6960.845910] Lustre: server umount lustre-OST0000 complete [ 6960.850175] LustreError: 179609:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6968.952470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.536318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6974.938689] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6975.970884] LustreError: 180108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6975.998173] LustreError: 180108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6981.098174] LustreError: 180109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6982.700897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6983.125761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6987.149472] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6993.803596] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6999.673326] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7005.955332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.180977] LustreError: 181923:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7006.198601] Lustre: 181923:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.130@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7006.356422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7012.790805] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7019.690788] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7024.096530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7024.108481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.127136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7026.156778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7026.159441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7026.177514] Lustre: Skipped 1 previous similar message [ 7026.191597] Lustre: Skipped 1 previous similar message [ 7031.265391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7031.271198] Lustre: Skipped 3 previous similar messages [ 7036.383114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7036.386666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7036.397093] Lustre: Skipped 2 previous similar messages [ 7036.501354] Lustre: server umount lustre-MDT0000 complete [ 7039.405429] LustreError: 180089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685114 with bad export cookie 8476076677596770464 [ 7039.418833] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7039.658496] Lustre: server umount lustre-MDT0001 complete [ 7052.517754] Lustre: server umount lustre-OST0000 complete [ 7060.550534] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 7069.291682] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7100.266469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7100.500732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7100.532138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7100.608233] Lustre: lustre-MDT0000: new disk, initializing [ 7100.669437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7100.684156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7104.284735] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7113.061946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7113.159208] Lustre: 185451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7113.200927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7113.204368] Lustre: Skipped 1 previous similar message [ 7113.253598] Lustre: lustre-MDT0001: new disk, initializing [ 7113.360052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7113.424555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7113.437399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7117.326895] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7125.421209] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7130.106856] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7136.052705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7136.330114] Lustre: lustre-OST0000: new disk, initializing [ 7136.335528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7136.342759] Lustre: 187418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7136.433731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7142.077331] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7146.613081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7146.656386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7146.830233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7153.055119] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7160.719551] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7161.021881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7166.092500] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7166.343225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7172.579101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7172.584187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.592523] LustreError: Skipped 1 previous similar message [ 7172.594184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7172.612151] Lustre: Skipped 1 previous similar message [ 7172.629725] Lustre: Skipped 1 previous similar message [ 7174.549788] Lustre: server umount lustre-OST0000 complete [ 7180.256426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7180.269050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7182.817269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7182.832200] Lustre: Skipped 2 previous similar messages [ 7184.941439] Lustre: server umount lustre-MDT0000 complete [ 7187.943536] LustreError: 185462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7187.964544] LustreError: 185462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7189.039200] LustreError: 187417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685263 with bad export cookie 8476076677596771192 [ 7189.046780] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7189.052369] LustreError: 187417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7189.336258] Lustre: server umount lustre-MDT0001 complete [ 7197.198606] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 05:01:10 (1788685270) [ 7212.694806] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 7215.195825] Key type lgssc unregistered [ 7215.472125] LNet: 190372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7215.481217] LNetError: 190372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7215.496788] LNet: Removed LNI 192.168.206.130@tcp [ 7216.270079] Key type .llcrypt unregistered [ 7216.272593] Key type ._llcrypt unregistered [ 7231.964047] Key type ._llcrypt registered [ 7231.965832] Key type .llcrypt registered [ 7232.076972] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7232.793867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7232.812054] alg: No test for adler32 (adler32-zlib) [ 7233.836254] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 7234.075258] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 7235.727245] Key type lgssc registered [ 7236.676700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7243.369021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7243.400513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.843681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7248.662392] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7255.017559] LustreError: 191584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7255.033909] LustreError: 191584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7256.325353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.690827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7260.590950] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7267.303434] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7272.979627] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7279.747029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7279.895216] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7279.897844] LustreError: 193449:0:(llog.c:1696:llog_backup()) MGC192.168.206.130@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7279.904385] Lustre: 193449:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.130@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7279.980528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7285.095509] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7291.530409] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7295.458383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7295.466863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7295.469806] Lustre: Skipped 2 previous similar messages [ 7295.474980] Lustre: Skipped 1 previous similar message [ 7299.707488] Lustre: server umount lustre-MDT0000 complete [ 7300.578832] LustreError: 191579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7300.612696] LustreError: 191579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7303.398537] LustreError: 192321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685378 with bad export cookie 7332946979100692206 [ 7303.412625] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7303.415545] LustreError: 192321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7303.759809] Lustre: server umount lustre-MDT0001 complete [ 7317.832205] Lustre: server umount lustre-OST0000 complete [ 7326.168788] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 7334.619974] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7366.804612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.015602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7367.046226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7367.120217] Lustre: lustre-MDT0000: new disk, initializing [ 7367.206319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.233865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7371.050937] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7380.549848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.672488] Lustre: 196977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7380.705912] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7380.707907] Lustre: Skipped 1 previous similar message [ 7380.710586] LustreError: 197760:0:(llog.c:1696:llog_backup()) MGC192.168.206.130@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7380.721013] LustreError: 197760:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7380.726626] Lustre: 197760:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.130@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7380.742980] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7380.750241] Lustre: Skipped 1 previous similar message [ 7380.851861] Lustre: lustre-MDT0001: new disk, initializing [ 7380.955644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7380.993342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7381.005610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7385.299688] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7394.433505] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7400.577671] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7407.772431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7408.109252] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7408.112129] Lustre: Skipped 1 previous similar message [ 7408.119288] LustreError: 198920:0:(llog.c:1696:llog_backup()) MGC192.168.206.130@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7408.132940] LustreError: 198920:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7408.138868] Lustre: 198920:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.130@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7408.152720] Lustre: 198920:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7408.179838] Lustre: lustre-OST0000: new disk, initializing [ 7408.186546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7408.196107] Lustre: 198944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7408.303869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.104398] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7417.393898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7417.400946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7417.488909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7424.357361] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7428.211952] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7428.378757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7432.143843] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7432.325770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7438.306360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.310098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7438.310193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7438.320229] Lustre: Skipped 1 previous similar message [ 7440.523607] Lustre: server umount lustre-OST0000 complete [ 7447.520287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7447.527985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7447.552259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7447.564302] Lustre: Skipped 1 previous similar message [ 7450.631302] Lustre: server umount lustre-MDT0000 complete [ 7453.666946] LustreError: 198952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.682944] LustreError: 198952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7454.336799] LustreError: 196969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685529 with bad export cookie 7332946979100692927 [ 7454.346906] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7454.351619] LustreError: 196969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7454.692949] Lustre: server umount lustre-MDT0001 complete [ 7463.692677] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 05:05:36 (1788685536) [ 7482.570259] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7492.204157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.601233] LustreError: 202069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.644525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7496.932577] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7497.710427] LustreError: 202070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7504.154773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7508.336359] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7510.851216] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7517.221593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7523.054609] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7530.232332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7530.365344] Lustre: lustre-OST0001: new disk, initializing [ 7530.374922] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7530.385711] Lustre: 204375:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7530.498307] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7530.502913] Lustre: Skipped 2 previous similar messages [ 7535.753788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7535.770357] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7535.930933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7537.777882] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7549.332903] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7559.682532] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7561.781310] Lustre: DEBUG MARKER: ADD OST3 [ 7566.155942] loop: module loaded [ 7566.209572] loop0: detected capacity change from 0 to 204800000 [ 7566.307861] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7566.312384] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7566.318807] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7571.270593] loop0: detected capacity change from 0 to 204800000 [ 7573.316610] Lustre: DEBUG MARKER: STOP OST3 [ 7577.702140] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7577.974325] Lustre: lustre-OST0003: new disk, initializing [ 7577.989919] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7578.007664] Lustre: 206317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7579.530186] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7579.732527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7579.757086] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7579.820268] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7579.957878] Lustre: Failing over lustre-OST0003 [ 7580.060457] Lustre: server umount lustre-OST0003 complete [ 7581.602303] LustreError: 206425:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7584.754345] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7584.754755] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7584.779266] LustreError: 203555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7584.789161] Lustre: Skipped 2 previous similar messages [ 7584.826674] LustreError: 203555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7585.598166] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7596.059635] Lustre: DEBUG MARKER: START OST3 again [ 7603.576548] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7603.895380] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7603.900544] Lustre: Skipped 1 previous similar message [ 7603.910807] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7605.208803] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7605.646550] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7605.652815] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7610.387382] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7630.822661] Lustre: Failing over lustre-OST0003 [ 7630.903392] Lustre: server umount lustre-OST0003 complete [ 7631.332884] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7631.334139] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.339209] LustreError: Skipped 1 previous similar message [ 7631.343237] LustreError: 204670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7631.354374] Lustre: Skipped 1 previous similar message [ 7631.361836] LustreError: 204670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7647.715492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7647.721325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.732190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7647.739919] Lustre: Skipped 2 previous similar messages [ 7653.463107] Lustre: server umount lustre-MDT0000 complete [ 7656.860727] LustreError: 205204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685731 with bad export cookie 7332946979100693809 [ 7656.863420] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7656.871526] LustreError: 205204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7660.522298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7660.537139] Lustre: Skipped 5 previous similar messages [ 7666.656997] LustreError: 204644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7666.677635] LustreError: 204644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 7671.263118] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7671.524371] Lustre: server umount lustre-MDT0001 complete [ 7675.411683] Lustre: server umount lustre-OST0000 complete [ 7679.271591] Lustre: server umount lustre-OST0001 complete [ 7685.343688] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 7693.730962] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7729.429370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.749247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7729.786982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7729.864460] Lustre: lustre-MDT0000: new disk, initializing [ 7729.964794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7734.752515] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7747.126242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.291859] Lustre: 211840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7747.407776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7747.417838] Lustre: Skipped 1 previous similar message [ 7747.593761] Lustre: lustre-MDT0001: new disk, initializing [ 7747.732870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7747.743124] Lustre: Skipped 1 previous similar message [ 7747.783811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7747.800275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7753.452693] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7762.451490] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7767.824263] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7774.773541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7775.154661] Lustre: 213810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7776.957170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7777.027145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7781.405645] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7791.144767] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7795.838729] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7796.036455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7800.497215] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7800.762934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7802.862959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7802.864124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7802.883897] Lustre: Skipped 5 previous similar messages [ 7802.887983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7802.887994] Lustre: Skipped 4 previous similar messages [ 7802.922567] LustreError: Skipped 1 previous similar message [ 7809.031270] Lustre: server umount lustre-OST0000 complete [ 7818.670920] Lustre: server umount lustre-MDT0000 complete [ 7822.859110] LustreError: 211834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788685897 with bad export cookie 7332946979100696105 [ 7822.866664] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7822.873768] LustreError: 211834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7823.376323] Lustre: server umount lustre-MDT0001 complete [ 7828.345986] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 05:11:40 (1788685900) [ 7843.085736] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 7853.273842] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7885.124306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7885.316920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7885.359245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7885.362804] Lustre: Skipped 1 previous similar message [ 7885.451777] Lustre: lustre-MDT0000: new disk, initializing [ 7885.454938] Lustre: Skipped 1 previous similar message [ 7885.540863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7885.547870] Lustre: Skipped 1 previous similar message [ 7889.230755] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7899.021680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7899.152609] Lustre: 218417:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7899.546260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7903.703522] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7912.930780] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7918.427967] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7923.645283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7923.983789] Lustre: 220386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7925.886169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7929.349301] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7938.265489] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7942.278861] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7942.427576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7945.906588] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7946.070329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7951.336068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7951.344075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7951.365734] Lustre: Skipped 4 previous similar messages [ 7951.383384] Lustre: Skipped 8 previous similar messages [ 7953.877227] Lustre: server umount lustre-OST0000 complete [ 7961.057152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7971.295139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7971.808740] LustreError: 218428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.842407] LustreError: 218428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7973.525607] Lustre: server umount lustre-MDT0000 complete [ 7977.310555] LustreError: 218408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686052 with bad export cookie 7332946979100696987 [ 7977.320807] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7977.641694] Lustre: server umount lustre-MDT0001 complete [ 7986.218251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7990.611105] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7998.119968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8002.722504] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8009.680235] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8014.051830] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8019.732622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8020.041533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8020.044939] Lustre: Skipped 6 previous similar messages [ 8025.936816] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8032.950277] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8045.025706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8045.031669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.039807] Lustre: Skipped 2 previous similar messages [ 8045.044241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8045.050288] Lustre: Skipped 4 previous similar messages [ 8048.386726] Lustre: server umount lustre-OST0000 complete [ 8057.534596] Lustre: server umount lustre-MDT0000 complete [ 8060.697954] LustreError: 222585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686135 with bad export cookie 7332946979100697869 [ 8060.708596] LustreError: 222585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8060.717442] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8061.075467] Lustre: server umount lustre-MDT0001 complete [ 8171.531125] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 8173.939886] Key type lgssc unregistered [ 8174.173638] LNet: 226266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8174.181749] LNetError: 226266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8174.196751] LNet: Removed LNI 192.168.206.130@tcp [ 8174.910176] Key type .llcrypt unregistered [ 8174.914106] Key type ._llcrypt unregistered [ 8195.916327] Key type ._llcrypt registered [ 8195.919058] Key type .llcrypt registered [ 8196.083729] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 8209.880947] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 8210.643039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8210.830607] alg: No test for adler32 (adler32-zlib) [ 8211.819725] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 8212.002348] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 8213.655145] Key type lgssc registered [ 8214.808309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8247.896992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8247.944613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8249.113827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8249.137464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8249.216220] Lustre: lustre-MDT0000: new disk, initializing [ 8249.297198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8249.311720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8253.414488] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8262.925208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8263.041092] Lustre: 229613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8263.108766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8263.114853] Lustre: Skipped 1 previous similar message [ 8263.271102] Lustre: lustre-MDT0001: new disk, initializing [ 8263.384643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8263.420445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8263.441410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8268.129092] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8278.135745] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8283.640448] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8290.907987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.315640] Lustre: lustre-OST0000: new disk, initializing [ 8291.319392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8291.328399] Lustre: 231581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8291.434269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.160254] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8299.570362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8299.575863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8299.665238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8306.034660] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8309.860278] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8310.035713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8314.230800] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8314.460699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8319.969748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8319.978798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8320.002941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8320.491103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8320.494976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8320.520840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8320.528740] Lustre: Skipped 1 previous similar message [ 8325.603601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8325.610642] Lustre: Skipped 1 previous similar message [ 8330.207116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8330.551137] Lustre: server umount lustre-OST0000 complete [ 8335.328170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8335.341322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8335.368031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8340.959430] Lustre: server umount lustre-MDT0000 complete [ 8345.030806] LustreError: 230409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686419 with bad export cookie 14158240924166139680 [ 8345.031742] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8345.043432] LustreError: 230409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8345.302719] Lustre: server umount lustre-MDT0001 complete [ 8367.862212] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 8376.059550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.371300] LustreError: 234539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.384238] LustreError: 234539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8376.408799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8380.166550] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8381.410736] LustreError: 234540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8386.531931] LustreError: 234539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8387.354305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8387.582311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8391.348104] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8394.158656] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8400.873606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8401.283536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.460668] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8417.073683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8417.200387] Lustre: lustre-OST0001: new disk, initializing [ 8417.203760] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8417.207843] Lustre: 236844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8417.303633] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8423.001419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8423.006810] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8423.096661] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8423.162904] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8433.518629] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8438.632098] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8450.444394] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 05:22:03 (1788686523) [ 8452.085578] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8452.087969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8452.094514] Lustre: Skipped 2 previous similar messages [ 8452.102400] Lustre: Skipped 1 previous similar message [ 8457.195839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.210039] Lustre: Skipped 1 previous similar message [ 8462.309591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8462.316724] Lustre: Skipped 8 previous similar messages [ 8466.399219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8466.518948] Lustre: server umount lustre-MDT0000 complete [ 8467.425407] LustreError: 234539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8467.440059] LustreError: 234539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8468.941325] LustreError: 234519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686543 with bad export cookie 14158240924166140569 [ 8468.948134] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8468.952321] LustreError: 234519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8469.134326] Lustre: server umount lustre-MDT0001 complete [ 8481.834552] Lustre: server umount lustre-OST0000 complete [ 8495.247287] Lustre: server umount lustre-OST0001 complete [ 8501.581791] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 8511.553578] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 8546.445253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8546.796744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8546.894083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8546.965293] Lustre: lustre-MDT0000: new disk, initializing [ 8547.030557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8547.055918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8551.568706] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8561.415486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8561.529248] Lustre: 240791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8561.580591] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8561.585689] Lustre: Skipped 1 previous similar message [ 8561.644501] Lustre: lustre-MDT0001: new disk, initializing [ 8561.806291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8561.826073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8566.493378] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8576.623781] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8582.841997] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8590.114059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8590.421454] Lustre: lustre-OST0000: new disk, initializing [ 8590.427880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8590.436665] Lustre: 242762:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8591.524685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8591.529591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8591.649663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8597.072708] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8606.693460] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8612.260963] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8612.464610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8617.828253] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8618.013242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8622.560790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8622.562926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8622.582818] LustreError: Skipped 1 previous similar message [ 8622.620238] Lustre: Skipped 1 previous similar message [ 8622.624719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.335133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8634.570344] Lustre: server umount lustre-OST0000 complete [ 8638.434055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8643.046490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8643.068314] Lustre: Skipped 1 previous similar message [ 8644.733017] Lustre: server umount lustre-MDT0000 complete [ 8648.156201] LustreError: 241585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686722 with bad export cookie 14158240924166142704 [ 8648.163815] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8648.177647] LustreError: 241585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8648.506492] Lustre: server umount lustre-MDT0001 complete [ 8657.168232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8657.837557] LustreError: 244978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8657.982721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8657.989882] Lustre: Skipped 2 previous similar messages [ 8663.261786] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8668.131815] LustreError: 244978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8668.152589] LustreError: 244978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8673.542032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.719762] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8687.244625] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8692.941585] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8700.551705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8707.935520] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8715.805534] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8861.664127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8861.672867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8861.678369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8861.689923] Lustre: Skipped 1 previous similar message [ 8861.695232] Lustre: Skipped 8 previous similar messages [ 8866.792358] LustreError: 246804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8866.808818] LustreError: 246804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8866.906373] Lustre: server umount lustre-OST0000 complete [ 8877.448568] Lustre: server umount lustre-MDT0000 complete [ 8881.385150] LustreError: 247601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788686956 with bad export cookie 14158240924166143586 [ 8881.392542] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8881.396839] LustreError: 247601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8881.719686] Lustre: server umount lustre-MDT0001 complete [ 8983.367532] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 8985.698667] Key type lgssc unregistered [ 8985.964334] LNet: 248588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8985.969027] LNetError: 248588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8985.984844] LNet: Removed LNI 192.168.206.130@tcp [ 8986.854773] Key type .llcrypt unregistered [ 8986.858940] Key type ._llcrypt unregistered [ 9009.478931] Key type ._llcrypt registered [ 9009.480795] Key type .llcrypt registered [ 9009.538831] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 9024.933607] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9025.588076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9025.598422] alg: No test for adler32 (adler32-zlib) [ 9026.621840] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [ 9026.796262] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 9028.439228] Key type lgssc registered [ 9029.608136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9075.826567] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9084.365338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9084.393352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9085.646477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9085.676503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9085.784124] Lustre: lustre-MDT0000: new disk, initializing [ 9085.859717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9085.868610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9089.651289] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9099.613436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9099.757706] Lustre: 252657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9099.802124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9099.805958] Lustre: Skipped 1 previous similar message [ 9099.897475] Lustre: lustre-MDT0001: new disk, initializing [ 9100.020707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9100.069414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9100.098079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9104.905761] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9109.522842] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9116.082253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9116.403824] Lustre: lustre-OST0000: new disk, initializing [ 9116.412342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9116.419978] Lustre: 254286:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9116.508825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9119.254501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9119.267778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9119.412523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9123.450914] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9134.061842] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9134.271130] Lustre: lustre-OST0001: new disk, initializing [ 9134.275532] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9134.281918] Lustre: 255157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9134.429192] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9140.554944] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9141.839682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9141.851663] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9141.992045] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9150.802875] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9162.385655] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9177.058243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9177.075488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9177.088522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9178.085605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9178.086402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9178.100063] Lustre: Skipped 1 previous similar message [ 9179.859818] Lustre: server umount lustre-MDT0000 complete [ 9183.834781] LustreError: 254285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687258 with bad export cookie 9706640217122971605 [ 9183.837636] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9183.846981] LustreError: 254285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9184.194061] Lustre: server umount lustre-MDT0001 complete [ 9197.685853] Lustre: server umount lustre-OST0000 complete [ 9211.855783] Lustre: server umount lustre-OST0001 complete [ 9224.634705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9233.023171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9241.108778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9251.771553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9270.409474] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9280.298221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9280.313579] Lustre: lustre-MDT0000: reset Object Index mappings [ 9280.776520] LustreError: 260169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9280.802816] LustreError: 260169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9280.883938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9285.711970] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9286.114680] LustreError: 260170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9290.215993] LustreError: 260169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9293.467444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9293.482392] Lustre: lustre-MDT0001: reset Object Index mappings [ 9293.813274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9297.807894] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9300.897604] Lustre: 261310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9307.909443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9314.851639] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9318.438119] LustreError: 261667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9323.506346] LustreError: 261666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9324.795473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9325.066258] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9325.071757] Lustre: Skipped 1 previous similar message [ 9326.105104] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9326.106476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9330.912383] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9338.353806] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9341.957278] Lustre: 263183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9346.534519] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9346.536732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9346.547712] Lustre: Skipped 3 previous similar messages [ 9346.565143] Lustre: Skipped 6 previous similar messages [ 9351.658577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9351.664700] Lustre: Skipped 3 previous similar messages [ 9352.120160] Lustre: server umount lustre-MDT0000 complete [ 9355.700918] LustreError: 260151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687430 with bad export cookie 9706640217122978402 [ 9355.716377] LustreError: 260151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9355.717226] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9356.135191] Lustre: server umount lustre-MDT0001 complete [ 9369.929491] Lustre: server umount lustre-OST0000 complete [ 9382.528034] Lustre: server umount lustre-OST0001 complete [ 9403.784411] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 05:37:56 (1788687476) [ 9420.182320] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 9432.097823] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9470.669457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9470.863751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9470.882679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9470.961681] Lustre: lustre-MDT0000: new disk, initializing [ 9471.092543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9471.130753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9474.946368] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9487.127156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9487.242888] Lustre: 267437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9487.321812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9487.331726] Lustre: Skipped 1 previous similar message [ 9487.426589] Lustre: lustre-MDT0001: new disk, initializing [ 9487.566500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9487.581030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9493.247084] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9503.004897] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9509.121372] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9515.470160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9515.685681] Lustre: lustre-OST0000: new disk, initializing [ 9515.690066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9515.703211] Lustre: 269406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9516.968525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9516.978443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9517.140143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9522.944963] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9532.739639] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9538.170176] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9538.439172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9544.370082] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9544.714545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9547.749652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9547.767240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9547.771722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9547.780877] Lustre: Skipped 2 previous similar messages [ 9547.799724] Lustre: Skipped 1 previous similar message [ 9553.509939] Lustre: server umount lustre-OST0000 complete [ 9559.520957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9559.538987] LustreError: Skipped 1 previous similar message [ 9559.544754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9559.568283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9559.573876] Lustre: Skipped 2 previous similar messages [ 9573.344578] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9573.537336] Lustre: server umount lustre-MDT0000 complete [ 9577.138423] LustreError: 267430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687651 with bad export cookie 9706640217122980495 [ 9577.140886] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9577.151674] LustreError: 267430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9577.404752] Lustre: server umount lustre-MDT0001 complete [ 9593.372412] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9602.862210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9603.264439] LustreError: 272343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9603.327440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9603.332368] Lustre: Skipped 2 previous similar messages [ 9607.164857] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9614.564533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9619.740682] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9623.169157] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9630.011035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9636.315443] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9643.946757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9644.127702] Lustre: lustre-OST0001: new disk, initializing [ 9644.133254] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9644.140892] Lustre: 274650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9645.602780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9645.618473] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9645.725681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9649.994761] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9658.981855] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9662.385899] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9667.076206] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9667.403918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9676.256719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9676.263940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9676.271170] Lustre: Skipped 1 previous similar message [ 9676.276581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9676.281271] Lustre: Skipped 6 previous similar messages [ 9682.313281] Lustre: server umount lustre-MDT0000 complete [ 9684.968543] LustreError: 272343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9684.978065] LustreError: 272343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9685.872434] LustreError: 274649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687760 with bad export cookie 9706640217122981370 [ 9685.875828] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9685.885319] LustreError: 274649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9686.161076] Lustre: server umount lustre-MDT0001 complete [ 9699.376576] Lustre: server umount lustre-OST0000 complete [ 9713.223800] Lustre: server umount lustre-OST0001 complete [ 9728.861537] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9738.851423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9739.238519] LustreError: 277996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9739.318948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9739.326431] Lustre: Skipped 3 previous similar messages [ 9743.760452] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9753.507345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9759.638350] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9762.928328] Lustre: 279136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9769.585711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9775.874954] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9783.859923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9785.174727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9785.207857] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9791.473321] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9800.305496] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9803.833135] Lustre: 281013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9809.116260] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9809.129785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9822.628772] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 05:44:55 (1788687895) [ 9841.120625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9841.131913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9841.147821] Lustre: Skipped 3 previous similar messages [ 9841.154293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9841.159340] Lustre: Skipped 4 previous similar messages [ 9844.880043] Lustre: server umount lustre-MDT0000 complete [ 9846.753662] LustreError: 277996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9846.770684] LustreError: 277996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9848.935627] LustreError: 277977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788687923 with bad export cookie 9706640217122987754 [ 9848.938184] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9848.946756] LustreError: 277977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9855.716517] Lustre: server umount lustre-MDT0001 complete [ 9866.417239] Lustre: server umount lustre-OST0000 complete [ 9870.753724] Lustre: server umount lustre-OST0001 complete [ 9877.446881] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 9886.552589] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9933.451248] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9944.030525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9944.305277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9944.341638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9944.471284] Lustre: lustre-MDT0000: new disk, initializing [ 9944.572155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9948.955663] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9960.145572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9960.262694] Lustre: 285551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9960.576833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9965.345587] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9977.878949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9978.266758] Lustre: 287184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9979.543355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9984.894783] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9995.066689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9995.176607] Lustre: 288054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9997.194141] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9997.211923] Lustre: Skipped 1 previous similar message [ 9997.369211] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10001.846398] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10011.014797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10014.561953] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10014.564667] Lustre: Skipped 1 previous similar message [10032.897930] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 05:48:25 (1788688105) [10037.216520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10037.228845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10037.249731] Lustre: Skipped 5 previous similar messages [10037.256099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10037.258501] Lustre: Skipped 7 previous similar messages [10037.798102] Lustre: server umount lustre-MDT0000 complete [10038.246087] LustreError: 285563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10038.271519] LustreError: 285563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10041.537306] LustreError: 285543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688116 with bad export cookie 9706640217122995678 [10041.540128] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10041.547909] LustreError: 285543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10041.932483] Lustre: server umount lustre-MDT0001 complete [10055.359946] Lustre: server umount lustre-OST0000 complete [10068.468512] Lustre: server umount lustre-OST0001 complete [10076.337286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10076.752791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10076.757761] Lustre: Skipped 7 previous similar messages [10080.361769] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10088.295439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10093.665424] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10102.261891] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10108.808200] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10117.081977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10124.824681] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10133.120606] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10155.002649] loop1: detected capacity change from 0 to 204800000 [10155.224086] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10155.246904] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10155.269080] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10160.803494] loop1: detected capacity change from 0 to 204800000 [10166.252119] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10166.395057] LustreError: 290289:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs89: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10166.406850] LustreError: 290289:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs89 [10166.412546] LustreError: 290289:0:(mgs_handler.c:517:mgs_target_reg()) fs89-MDT0001: Error checking target index: rc = -13 [10166.420363] LustreError: fs89-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10166.427548] LustreError: 294021:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -13 [10166.431539] LustreError: 294021:0:(tgt_mount.c:2134:server_put_super()) no obd fs89-MDT0001 [10166.434588] LustreError: 294021:0:(tgt_mount.c:116:server_deregister_mount()) fs89-MDT0001 not registered [10166.460313] Lustre: server umount fs89-MDT0001 complete [10166.464251] LustreError: 294021:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10175.479273] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10175.608985] Lustre: Setting parameter fs89-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs89-MDT0001 [10175.656375] Lustre: srv-fs89-MDT0001: No data found on store. Initialize space. [10175.659332] Lustre: Skipped 4 previous similar messages [10175.749315] Lustre: fs89-MDT0001: new disk, initializing [10175.753515] Lustre: Skipped 3 previous similar messages [10175.913746] Lustre: srv-fs89-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10175.920774] Lustre: 294488:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs89-MDT0001-mdtlov: can't create new llog: rc = -115 [10175.943233] LustreError: 294488:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs89-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10180.783255] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10187.518357] Lustre: server umount fs89-MDT0001 complete [10200.555898] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10201.842559] loop1: detected capacity change from 0 to 204800000 [10208.366368] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10209.204440] Lustre: 296059:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs89-MDT0001-mdtlov: can't create new llog: rc = -115 [10209.214757] LustreError: 296059:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs89-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [10214.844185] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10220.386956] Lustre: server umount fs89-MDT0001 complete [10231.288284] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10231.302405] Lustre: Skipped 5 previous similar messages [10231.305824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10231.324126] Lustre: Skipped 2 previous similar messages [10236.834297] Lustre: server umount lustre-MDT0000 complete [10245.666839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.884353] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10251.744089] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10251.771312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10259.816939] loop1: detected capacity change from 0 to 204800000 [10265.607189] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10266.028104] Lustre: 298506:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs89-MDT0001-mdtlov: can't create new llog: rc = -115 [10266.042673] LustreError: 298506:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs89-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10270.522618] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10275.513154] Lustre: server umount fs89-MDT0001 complete [10297.844834] LustreError: 292129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10297.884701] LustreError: 292129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 104 previous similar messages [10302.434410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10302.449467] LustreError: Skipped 1 previous similar message [10305.739510] Lustre: server umount lustre-OST0000 complete [10316.233824] Lustre: server umount lustre-MDT0000 complete [10320.194297] LustreError: 290281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688394 with bad export cookie 9706640217122998359 [10320.199410] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10320.211018] LustreError: 290281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10320.673280] Lustre: server umount lustre-MDT0001 complete [10332.468637] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [10335.108642] Key type lgssc unregistered [10335.416637] LNet: 300850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10335.424676] LNetError: 300850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10335.457604] LNet: Removed LNI 192.168.206.130@tcp [10336.235138] Key type .llcrypt unregistered [10336.236894] Key type ._llcrypt unregistered [10356.031619] Key type ._llcrypt registered [10356.034293] Key type .llcrypt registered [10356.144850] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [10357.634464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10357.669431] alg: No test for adler32 (adler32-zlib) [10358.786905] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [10359.090097] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [10360.807193] Key type lgssc registered [10361.998208] Lustre: Echo OBD driver; http://www.lustre.org/ [10370.030512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10370.057038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.474390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10376.452631] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10381.798066] LustreError: 302118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10381.808740] LustreError: 302118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10384.821265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10385.229227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10390.117884] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10398.587409] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10404.818635] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10411.626788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10412.114735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10417.194720] LustreError: 303948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10419.039871] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10422.249062] LustreError: 303947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10426.564242] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10427.370391] LustreError: 303949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10433.316765] LustreError: 304216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10433.336708] LustreError: 304216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10443.560552] LustreError: 304216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10443.578879] LustreError: 304216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10449.262956] loop1: detected capacity change from 0 to 204800000 [10449.456802] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10449.477759] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10449.494681] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10455.063902] loop1: detected capacity change from 0 to 204800000 [10460.144243] LustreError: 303947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10460.169327] LustreError: 303947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10460.596348] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10460.778883] Lustre: fs28317-OST0000: new disk, initializing [10460.781214] Lustre: srv-fs28317-OST0000: No data found on store. Initialize space. [10460.795986] Lustre: 305853:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10460.884387] Lustre: fs28317-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10467.036248] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10476.729059] Lustre: server umount fs28317-OST0000 complete [10492.531172] loop1: detected capacity change from 0 to 204800000 [10494.752518] LustreError: 304216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10494.795679] LustreError: 304216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [10495.977503] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10499.208818] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10499.626171] Lustre: fs28317-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10506.600340] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10511.338257] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10515.625445] loop2: detected capacity change from 0 to 204800000 [10515.882591] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10515.903303] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10515.953322] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10521.626756] loop2: detected capacity change from 0 to 204800000 [10527.666576] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10527.823069] LustreError: fs28317-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10527.831879] LustreError: fs28317-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10527.836362] LustreError: 308895:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -13 [10527.839864] LustreError: 308895:0:(tgt_mount.c:2134:server_put_super()) no obd fs28317-OST0001 [10527.844321] LustreError: 308895:0:(tgt_mount.c:116:server_deregister_mount()) fs28317-OST0001 not registered [10527.872928] Lustre: server umount fs28317-OST0001 complete [10527.875479] LustreError: 308895:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10538.292819] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10538.433669] Lustre: fs28317-OST0001: new disk, initializing [10538.444397] Lustre: srv-fs28317-OST0001: No data found on store. Initialize space. [10538.458905] Lustre: 309446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10538.547473] Lustre: fs28317-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10544.982783] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10553.318283] Lustre: server umount fs28317-OST0000 complete [10561.349430] LustreError: 303948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.371930] LustreError: 303948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10561.846803] Lustre: server umount fs28317-OST0001 complete [10574.032342] loop1: detected capacity change from 0 to 204800000 [10579.577166] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10579.849473] Lustre: fs28317-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10586.478389] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10593.416406] loop2: detected capacity change from 0 to 204800000 [10599.211760] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10607.265979] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10617.368582] Lustre: server umount fs28317-OST0000 complete [10627.725352] Lustre: server umount fs28317-OST0001 complete [10642.401745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10642.420473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10642.447795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10644.465243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10644.475212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10644.476446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10647.731325] Lustre: server umount lustre-OST0000 complete [10654.689186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10654.690685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10654.703240] Lustre: Skipped 1 previous similar message [10654.713986] Lustre: Skipped 1 previous similar message [10658.325405] Lustre: server umount lustre-MDT0000 complete [10662.320772] LustreError: 302098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688737 with bad export cookie 14955523782560052316 [10662.330714] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10662.337468] LustreError: 302098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10663.060881] Lustre: server umount lustre-MDT0001 complete [10674.926791] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [10677.810939] Key type lgssc unregistered [10678.083817] LNet: 315370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10678.096297] LNetError: 315370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10678.112707] LNet: Removed LNI 192.168.206.130@tcp [10678.925240] Key type .llcrypt unregistered [10678.931442] Key type ._llcrypt unregistered [10708.911958] Key type ._llcrypt registered [10708.916838] Key type .llcrypt registered [10709.051421] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [10726.997232] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [10728.166793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10728.399506] alg: No test for adler32 (adler32-zlib) [10729.706239] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [10729.997323] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [10731.687327] Key type lgssc registered [10732.896252] Lustre: Echo OBD driver; http://www.lustre.org/ [10776.191619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10776.212792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10777.511264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10777.557151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10777.635902] Lustre: lustre-MDT0000: new disk, initializing [10777.716190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10777.732589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10782.832277] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10794.829978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10794.907497] Lustre: 319224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10794.960202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10794.963052] Lustre: Skipped 1 previous similar message [10795.085411] Lustre: lustre-MDT0001: new disk, initializing [10795.197527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10795.239715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10795.254540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10799.909330] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10810.015888] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10818.103741] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10825.334613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10825.591445] Lustre: lustre-OST0000: new disk, initializing [10825.594692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10825.604873] Lustre: 321206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10825.684812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10831.883469] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10835.543549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10835.555908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10835.712850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10841.223694] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10843.578717] Lustre: Mounted lustre-client - version 2.17.58_4_g3b30273 [10844.345117] Lustre: Unmounted lustre-client [10853.868945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10853.883346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10853.887807] Lustre: Skipped 1 previous similar message [10858.981901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10858.995980] Lustre: Skipped 2 previous similar messages [10859.337720] Lustre: server umount lustre-OST0000 complete [10864.098791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10864.107516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10864.117414] Lustre: Skipped 1 previous similar message [10864.139113] Lustre: Skipped 1 previous similar message [10869.164514] Lustre: server umount lustre-MDT0000 complete [10872.796281] LustreError: 319216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788688947 with bad export cookie 15526373702023358864 [10872.810283] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10873.232665] Lustre: server umount lustre-MDT0001 complete [10883.161793] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [10885.637951] Key type lgssc unregistered [10899.233592] LNet: 323367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10899.276425] LNetError: 323367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10975.083910] LNet: Removed LNI 192.168.206.130@tcp [10976.403785] Key type .llcrypt unregistered [10976.407736] Key type ._llcrypt unregistered [10996.074569] Key type ._llcrypt registered [10996.080759] Key type .llcrypt registered [10996.201280] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [11013.492827] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [11015.009124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11015.024213] alg: No test for adler32 (adler32-zlib) [11016.060901] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [11016.306779] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [11018.007223] Key type lgssc registered [11019.005992] Lustre: Echo OBD driver; http://www.lustre.org/ [11051.621613] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 06:05:24 (1788689124) [11059.254408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11059.274357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11060.614432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11060.679054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11060.828938] Lustre: lustre-MDT0000: new disk, initializing [11061.004828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11061.065348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11066.729421] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11077.650926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11077.778831] Lustre: 326585:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11077.854675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11077.859960] Lustre: Skipped 1 previous similar message [11077.959600] Lustre: lustre-MDT0001: new disk, initializing [11078.104265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11078.179373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11078.193856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11082.828434] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11091.845792] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11097.484717] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11103.944416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11104.224674] Lustre: lustre-OST0000: new disk, initializing [11104.227826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11104.234875] Lustre: 328554:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11104.335591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11109.957914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11109.972532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11110.095224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11110.153945] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11118.872934] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11135.970207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11135.970716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11135.976224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11135.979495] LustreError: Skipped 1 previous similar message [11135.993349] Lustre: Skipped 1 previous similar message [11140.025598] Lustre: server umount lustre-OST0000 complete [11144.681835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11144.695165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11144.707066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11144.716164] Lustre: Skipped 1 previous similar message [11146.208735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11146.214043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11150.219306] Lustre: server umount lustre-MDT0000 complete [11153.693029] LustreError: 326576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689228 with bad export cookie 14585115157772757220 [11153.695531] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11153.701285] LustreError: 326576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11154.065245] Lustre: server umount lustre-MDT0001 complete [11164.832481] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11167.658690] Key type lgssc unregistered [11168.038780] LNet: 330463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11168.047029] LNetError: 330463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11168.071116] LNet: Removed LNI 192.168.206.130@tcp [11169.069833] Key type .llcrypt unregistered [11169.072069] Key type ._llcrypt unregistered [11190.777683] Key type ._llcrypt registered [11190.779725] Key type .llcrypt registered [11190.896411] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [11191.804243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11191.891338] alg: No test for adler32 (adler32-zlib) [11192.895619] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [11193.137430] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [11194.816266] Key type lgssc registered [11195.733964] Lustre: Echo OBD driver; http://www.lustre.org/ [11203.877984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11203.917733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11205.526664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11210.036685] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11215.843728] LustreError: 331733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11215.896156] LustreError: 331733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11217.951072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11218.247871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11222.593872] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11231.530552] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11239.042679] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11246.868801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11247.124838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11252.296650] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11259.084557] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11271.640472] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11273.218137] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11277.798358] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11277.811772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11277.821289] Lustre: Skipped 1 previous similar message [11277.835666] Lustre: Skipped 2 previous similar messages [11281.241398] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.30@tcp (stopping) [11282.913222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11282.933284] Lustre: Skipped 2 previous similar messages [11285.545453] Lustre: server umount lustre-MDT0000 complete [11286.312503] LustreError: 331729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.034874] LustreError: 331734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.060849] LustreError: 331734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11289.529370] LustreError: 331714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689364 with bad export cookie 7113756527899844316 [11289.534951] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11289.542866] LustreError: 331714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11291.420230] LustreError: 331730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.451453] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.30@tcp (stopping) [11293.168799] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11293.183946] Lustre: Skipped 1 previous similar message [11296.056578] Lustre: server umount lustre-MDT0001 complete [11300.045173] Lustre: server umount lustre-OST0000 complete [11308.336312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11312.787475] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11315.293439] Lustre: 335914:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.130@tcp [11317.388971] Lustre: server umount MGS complete [11326.433638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11326.723355] LustreError: 336486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11326.741523] LustreError: 336486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11326.805891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11331.253445] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11336.411436] LustreError: 336483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11336.437983] LustreError: 336483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11340.115907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11340.620445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11346.201868] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11355.647962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11356.026182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11362.619969] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11370.572547] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11372.387412] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11378.517985] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11380.134353] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11385.827693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11385.842272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11385.859083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11385.863201] Lustre: Skipped 2 previous similar messages [11386.858402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11387.445460] Lustre: server umount lustre-OST0000 complete [11391.969824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11391.975524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11391.980520] Lustre: Skipped 2 previous similar messages [11397.088275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11397.095171] Lustre: Skipped 4 previous similar messages [11405.279266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11405.482840] Lustre: server umount lustre-MDT0000 complete [11407.345339] LustreError: 336486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11407.362558] LustreError: 336486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11409.412170] LustreError: 336468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689484 with bad export cookie 7113756527899845156 [11409.415811] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11409.423228] LustreError: 336468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11409.853792] Lustre: server umount lustre-MDT0001 complete [11421.152933] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11424.282185] Key type lgssc unregistered [11424.615723] LNet: 339923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11424.628430] LNetError: 339923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11425.719103] LNet: Removed LNI 192.168.206.130@tcp [11426.744344] Key type .llcrypt unregistered [11426.748501] Key type ._llcrypt unregistered [11446.300522] Key type ._llcrypt registered [11446.302744] Key type .llcrypt registered [11446.492350] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [11462.166424] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [11463.052318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11463.119042] alg: No test for adler32 (adler32-zlib) [11464.198601] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [11464.466060] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [11466.159150] Key type lgssc registered [11467.301292] Lustre: Echo OBD driver; http://www.lustre.org/ [11504.830655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11504.842084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11506.043617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11506.093573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11506.227225] Lustre: lustre-MDT0000: new disk, initializing [11506.331957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11506.356653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11511.363680] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11521.717837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11521.850360] Lustre: 343085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11521.902926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11521.908040] Lustre: Skipped 1 previous similar message [11522.042970] Lustre: lustre-MDT0001: new disk, initializing [11522.148717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11522.199679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11522.210254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11526.963434] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11536.152656] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11542.153479] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11548.541661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11548.813295] Lustre: lustre-OST0000: new disk, initializing [11548.817747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11548.834744] Lustre: 345053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11548.936877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11553.837224] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11555.885155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11555.890554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11555.963165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11562.909397] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11568.785728] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11569.132492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11573.668222] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11573.887265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11576.288520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11576.301272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11576.337229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11576.801574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11576.816958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11576.842022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11581.937068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11581.948026] Lustre: Skipped 1 previous similar message [11582.406711] Lustre: server umount lustre-OST0000 complete [11587.044771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11587.060634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11587.067137] Lustre: Skipped 1 previous similar message [11591.616377] LustreError: 346592:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000b41f8076) refcount nonzero (1) after lock cleanup; forcing cleanup. [11591.658686] Lustre: server umount lustre-MDT0000 complete [11594.860962] LustreError: 343880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689669 with bad export cookie 9766274144497965165 [11594.870736] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11594.876623] LustreError: 343880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11595.202371] Lustre: server umount lustre-MDT0001 complete [11604.148852] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 06:14:36 (1788689676) [11619.531307] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [11629.113627] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [11664.858663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11665.070343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11665.089238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11665.202609] Lustre: lustre-MDT0000: new disk, initializing [11665.292863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11665.312555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11670.265502] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11681.875784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11682.036945] Lustre: 349808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11682.099839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11682.105584] Lustre: Skipped 1 previous similar message [11682.215850] Lustre: lustre-MDT0001: new disk, initializing [11682.365557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11682.414296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11682.421384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11687.025952] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11697.329903] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11703.890262] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11711.253818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11711.647841] Lustre: lustre-OST0000: new disk, initializing [11711.661523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11711.669916] Lustre: 351778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11711.805761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11718.301898] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11719.796923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11719.819300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11719.978600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11728.935610] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11766.242234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11766.255020] Lustre: Skipped 1 previous similar message [11766.255780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11766.256517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11769.716872] Lustre: server umount lustre-OST0000 complete [11774.438836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11774.447706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11774.482978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11774.493861] Lustre: Skipped 1 previous similar message [11780.248743] Lustre: server umount lustre-MDT0000 complete [11781.619107] LustreError: 349817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11781.645408] LustreError: 349817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11783.546151] LustreError: 350613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788689858 with bad export cookie 9766274144497966047 [11783.548553] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11783.556445] LustreError: 350613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11783.884675] Lustre: server umount lustre-MDT0001 complete [11794.689624] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11797.614705] Key type lgssc unregistered [11797.936201] LNet: 353951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11797.944430] LNetError: 353951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11797.974986] LNet: Removed LNI 192.168.206.130@tcp [11798.876543] Key type .llcrypt unregistered [11798.879549] Key type ._llcrypt unregistered [11889.065714] Key type ._llcrypt registered [11889.068665] Key type .llcrypt registered [11889.187456] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11889.528287] Key type .llcrypt unregistered [11889.531124] Key type ._llcrypt unregistered [11964.535990] Key type ._llcrypt registered [11964.539190] Key type .llcrypt registered [11964.635214] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11964.957717] Key type .llcrypt unregistered [11964.965621] Key type ._llcrypt unregistered [11994.755582] Key type ._llcrypt registered [11994.758326] Key type .llcrypt registered [11994.855359] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11995.184739] Key type .llcrypt unregistered [11995.189272] Key type ._llcrypt unregistered [12029.333812] Key type ._llcrypt registered [12029.335486] Key type .llcrypt registered [12029.413761] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12029.832333] Key type .llcrypt unregistered [12029.838867] Key type ._llcrypt unregistered [12101.828624] Key type ._llcrypt registered [12101.831801] Key type .llcrypt registered [12101.936353] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12102.192359] Key type .llcrypt unregistered [12102.195438] Key type ._llcrypt unregistered [12131.166843] Key type ._llcrypt registered [12131.168919] Key type .llcrypt registered [12131.272336] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12131.550444] Key type .llcrypt unregistered [12131.556548] Key type ._llcrypt unregistered [12161.395984] Key type ._llcrypt registered [12161.398494] Key type .llcrypt registered [12161.495697] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12161.774295] Key type .llcrypt unregistered [12161.776801] Key type ._llcrypt unregistered [12190.708670] Key type ._llcrypt registered [12190.712738] Key type .llcrypt registered [12190.820447] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12191.114384] Key type .llcrypt unregistered [12191.119829] Key type ._llcrypt unregistered [12216.651270] Key type ._llcrypt registered [12216.654439] Key type .llcrypt registered [12216.791906] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12217.083296] Key type .llcrypt unregistered [12217.085568] Key type ._llcrypt unregistered [12245.114460] Key type ._llcrypt registered [12245.118473] Key type .llcrypt registered [12245.250493] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12245.591420] Key type .llcrypt unregistered [12245.593767] Key type ._llcrypt unregistered [12304.640409] Key type ._llcrypt registered [12304.645568] Key type .llcrypt registered [12304.769769] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12305.082050] Key type .llcrypt unregistered [12305.088718] Key type ._llcrypt unregistered [12338.688415] Key type ._llcrypt registered [12338.692456] Key type .llcrypt registered [12338.802647] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12339.061947] Key type .llcrypt unregistered [12339.069214] Key type ._llcrypt unregistered [12371.401762] Key type ._llcrypt registered [12371.404285] Key type .llcrypt registered [12371.506304] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12371.760395] Key type .llcrypt unregistered [12371.762765] Key type ._llcrypt unregistered [12424.911502] Key type ._llcrypt registered [12424.913397] Key type .llcrypt registered [12425.011766] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [12441.771071] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [12442.557704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12442.591671] alg: No test for adler32 (adler32-zlib) [12443.720610] Lustre: Lustre: Build Version: 2.17.58_4_g3b30273 [12443.949881] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [12445.640674] Key type lgssc registered [12446.572756] Lustre: Echo OBD driver; http://www.lustre.org/ [12480.968318] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12480.998965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12482.303849] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12482.329914] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12482.405308] Lustre: tst250fs-MDT0000: new disk, initializing [12482.474920] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12482.493220] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12486.715342] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [12496.515791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12496.611529] Lustre: 374216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12496.660647] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12496.664363] Lustre: Skipped 1 previous similar message [12496.722871] Lustre: tst250fs-MDT0001: new disk, initializing [12496.834226] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12496.881845] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12496.895254] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12501.578480] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [12506.703751] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12513.483521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12513.782280] Lustre: tst250fs-OST0000: new disk, initializing [12513.788963] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12513.803962] Lustre: 375851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12513.904619] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12520.861742] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [12522.583586] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12522.593082] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12522.696473] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12530.637167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12530.838143] Lustre: tst250fs-OST0001: new disk, initializing [12530.841344] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12530.850069] Lustre: 376722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12530.972441] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12537.257713] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [12538.543605] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12538.555709] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12538.664789] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12542.186779] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12544.598540] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12544.605583] Lustre: Skipped 2 previous similar messages [12563.424376] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12563.435172] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12563.456415] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12564.474160] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12564.476199] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12567.752108] Lustre: server umount tst250fs-MDT0000 complete [12571.658527] LustreError: 375850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788690646 with bad export cookie 12986995730462525090 [12571.667240] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12571.669486] LustreError: 375850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12572.097765] Lustre: server umount tst250fs-MDT0001 complete [12586.874674] Lustre: server umount tst250fs-OST0000 complete [12590.629514] Lustre: 372442:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788690649/real 1788690649] req@ffff8f962f9dc700 x1875577948667904/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788690665 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12590.667155] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12590.683758] Lustre: Skipped 2 previous similar messages [12590.912075] Lustre: server umount tst250fs-OST0001 complete [12609.110229] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [12617.478995] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [12645.848120] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 06:31:58 (1788690718) [12647.463035] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12663.304372] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [12673.656767] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [12708.895327] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12471 sec ========= 06:33:01 (1788690781) [12710.757825] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:33:03 (1788690783) === [12715.068781] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:33:07 (1788690787) === [12755.044614] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12757.947824] Key type lgssc unregistered [12758.189829] LNet: 385846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12758.196289] LNetError: 385846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12758.207299] LNet: Removed LNI 192.168.206.130@tcp [12758.941163] Key type .llcrypt unregistered [12758.943310] Key type ._llcrypt unregistered