[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481843565 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003237] x2apic enabled [ 0.004014] Switched APIC routing to physical x2apic. [ 0.005024] kvm-guest: setup PV IPIs [ 0.008550] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.011185] LSM: Security Framework initializing [ 0.012095] Yama: becoming mindful. [ 0.013057] SELinux: Initializing. [ 0.014105] *** VALIDATE selinux *** [ 0.022781] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027208] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028215] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030057] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032145] *** VALIDATE tmpfs *** [ 0.034186] *** VALIDATE proc *** [ 0.036077] *** VALIDATE cgroup *** [ 0.037012] *** VALIDATE cgroup2 *** [ 0.038281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039185] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041037] Spectre V2 : User space: Vulnerable [ 0.042011] Speculative Store Bypass: Vulnerable [ 0.045529] debug: unmapping init [mem 0xffffffff89059000-0xffffffff89060fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048810] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049031] ... version: 2 [ 0.050020] ... bit width: 48 [ 0.051020] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053023] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055017] ... event mask: 000000070000000f [ 0.057265] rcu: Hierarchical SRCU implementation. [ 0.059726] smp: Bringing up secondary CPUs ... [ 0.060581] x86: Booting SMP configuration: [ 0.061036] .... node #0, CPUs: #1 #2 #3 [ 0.065109] smp: Brought up 1 node, 4 CPUs [ 0.067025] smpboot: Max logical packages: 1 [ 0.068020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.254151] node 0 deferred pages initialised in 182ms [ 0.258161] devtmpfs: initialized [ 0.259287] x86/mm: Memory block size: 128MB [ 0.261941] gcov: version magic: 0x41383552 [ 0.263411] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.267131] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.269768] pinctrl core: initialized pinctrl subsystem [ 0.272192] [ 0.272755] ************************************************************* [ 0.275016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.278015] ** ** [ 0.280013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.282014] ** ** [ 0.284012] ** This means that this kernel is built to expose internal ** [ 0.285009] ** IOMMU data structures, which may compromise security on ** [ 0.288013] ** your system. ** [ 0.290013] ** ** [ 0.292016] ** If you see this message and you are not debugging the ** [ 0.294013] ** kernel, report this immediately to your vendor! ** [ 0.296018] ** ** [ 0.299019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.301014] ************************************************************* [ 0.304635] NET: Registered protocol family 16 [ 0.306434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.308066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.310059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.315053] cpuidle: using governor menu [ 0.316842] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.319838] PCI: Using configuration type 1 for base access [ 0.322157] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.332144] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.335076] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.339101] cryptd: max_cpu_qlen set to 1000 [ 0.342743] ACPI: Added _OSI(Module Device) [ 0.343000] ACPI: Added _OSI(Processor Device) [ 0.347028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.349019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.353000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.360574] ACPI: Interpreter enabled [ 0.362066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.363014] ACPI: Using IOAPIC for interrupt routing [ 0.365144] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.369482] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.380214] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.382062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.384029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.387118] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.392522] acpiphp: Slot [2] registered [ 0.394217] acpiphp: Slot [5] registered [ 0.396219] acpiphp: Slot [6] registered [ 0.398245] acpiphp: Slot [7] registered [ 0.399168] acpiphp: Slot [8] registered [ 0.400136] acpiphp: Slot [9] registered [ 0.402150] acpiphp: Slot [10] registered [ 0.404202] acpiphp: Slot [3] registered [ 0.406142] acpiphp: Slot [4] registered [ 0.407180] acpiphp: Slot [11] registered [ 0.409118] acpiphp: Slot [12] registered [ 0.410119] acpiphp: Slot [13] registered [ 0.412128] acpiphp: Slot [14] registered [ 0.414117] acpiphp: Slot [15] registered [ 0.415210] acpiphp: Slot [16] registered [ 0.417126] acpiphp: Slot [17] registered [ 0.418119] acpiphp: Slot [18] registered [ 0.420113] acpiphp: Slot [19] registered [ 0.422119] acpiphp: Slot [20] registered [ 0.423117] acpiphp: Slot [21] registered [ 0.425117] acpiphp: Slot [22] registered [ 0.427146] acpiphp: Slot [23] registered [ 0.428119] acpiphp: Slot [24] registered [ 0.430135] acpiphp: Slot [25] registered [ 0.431101] acpiphp: Slot [26] registered [ 0.433124] acpiphp: Slot [27] registered [ 0.435227] acpiphp: Slot [28] registered [ 0.437212] acpiphp: Slot [29] registered [ 0.439194] acpiphp: Slot [30] registered [ 0.440106] acpiphp: Slot [31] registered [ 0.442089] PCI host bridge to bus 0000:00 [ 0.444025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.446025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.448027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.450038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.453026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.455029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.457205] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.461073] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.464194] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.474952] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.480077] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.483023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.485017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.488022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.490481] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.493850] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.496047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.499867] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.506031] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.519883] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.524015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.531303] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.538018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.545019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.563022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.575536] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.582018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.590024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.603022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.613000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.621020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.632024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.661022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.673486] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.680017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.693021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.714022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.729866] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.738021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.744025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.761021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.772134] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.782025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.790023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.810021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.823776] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.826424] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.828439] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.831586] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.835364] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.841057] iommu: Default domain type: Passthrough [ 0.844632] SCSI subsystem initialized [ 0.846246] ACPI: bus type USB registered [ 0.847154] usbcore: registered new interface driver usbfs [ 0.849129] usbcore: registered new interface driver hub [ 0.851119] usbcore: registered new device driver usb [ 0.853220] pps_core: LinuxPPS API ver. 1 registered [ 0.855027] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.859112] PTP clock support registered [ 0.861060] EDAC MC: Ver: 3.0.0 [ 0.864264] PCI: Using ACPI for IRQ routing [ 0.867121] NetLabel: Initializing [ 0.868017] NetLabel: domain hash size = 128 [ 0.870016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.872111] NetLabel: unlabeled traffic allowed by default [ 0.874382] vgaarb: loaded [ 0.876357] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.878020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.882511] clocksource: Switched to clocksource kvm-clock [ 0.999775] VFS: Disk quotas dquot_6.6.0 [ 1.001578] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.004437] *** VALIDATE ramfs *** [ 1.005913] *** VALIDATE hugetlbfs *** [ 1.007722] pnp: PnP ACPI init [ 1.010351] pnp: PnP ACPI: found 6 devices [ 1.030796] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.035582] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.037759] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.039954] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.042866] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.045922] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.049356] NET: Registered protocol family 2 [ 1.052486] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.058599] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.062510] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.068275] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.071881] TCP: Hash tables configured (established 65536 bind 65536) [ 1.075034] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.078373] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.081426] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.084671] NET: Registered protocol family 1 [ 1.088756] RPC: Registered named UNIX socket transport module. [ 1.090797] RPC: Registered udp transport module. [ 1.092668] RPC: Registered tcp transport module. [ 1.094460] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.097081] NET: Registered protocol family 44 [ 1.098710] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.100930] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.103167] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.105649] PCI: CLS 0 bytes, default 64 [ 1.108550] Unpacking initramfs... [ 2.607504] debug: unmapping init [mem 0xffff91607cc54000-0xffff91607ffbffff] [ 2.614353] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.616858] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.621551] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.117753] Initialise system trusted keyrings [ 3.119429] Key type blacklist registered [ 3.121761] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.133970] zbud: loaded [ 3.137731] *** VALIDATE nfs *** [ 3.139289] *** VALIDATE nfs4 *** [ 3.141210] pstore: using deflate compression [ 3.148426] Platform Keyring initialized [ 3.276724] NET: Registered protocol family 38 [ 3.278833] Key type asymmetric registered [ 3.280545] Asymmetric key parser 'x509' registered [ 3.281929] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.286192] io scheduler mq-deadline registered [ 3.287897] io scheduler kyber registered [ 3.289891] io scheduler bfq registered [ 3.292161] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.295601] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.298657] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.301762] ACPI: Power Button [PWRF] [ 3.307525] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.314249] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.335651] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.346943] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.373163] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.402347] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.433633] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.439196] Non-volatile memory driver v1.3 [ 3.441100] Linux agpgart interface v0.103 [ 3.483231] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.488331] vda: detected capacity change from 0 to 76677120 [ 3.505689] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.509077] vdb: detected capacity change from 0 to 1073741824 [ 3.530797] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.534145] vdc: detected capacity change from 0 to 2621440000 [ 3.549183] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.552488] vdd: detected capacity change from 0 to 2621440000 [ 3.571674] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.574664] vde: detected capacity change from 0 to 4294967296 [ 3.594605] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.597894] vdf: detected capacity change from 0 to 4294967296 [ 3.613895] libphy: Fixed MDIO Bus: probed [ 3.621826] usbcore: registered new interface driver usbserial_generic [ 3.624915] usbserial: USB Serial support registered for generic [ 3.627393] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.631709] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.633790] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.636668] mousedev: PS/2 mouse device common for all mice [ 3.640683] rtc_cmos 00:05: RTC can wake from S4 [ 3.644391] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.649244] rtc_cmos 00:05: registered as rtc0 [ 3.654099] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.657208] intel_pstate: CPU model not supported [ 3.660733] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.664485] hid: raw HID events driver (C) Jiri Kosina [ 3.673576] usbcore: registered new interface driver usbhid [ 3.675686] usbhid: USB HID core driver [ 3.679353] drop_monitor: Initializing network drop monitor service [ 3.679753] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.682195] Initializing XFRM netlink socket [ 3.688693] NET: Registered protocol family 10 [ 3.691964] Segment Routing with IPv6 [ 3.693684] NET: Registered protocol family 17 [ 3.695877] mpls_gso: MPLS GSO support [ 3.701802] RAS: Correctable Errors collector initialized. [ 3.704018] AVX version of gcm_enc/dec engaged. [ 3.705869] AES CTR mode by8 optimization enabled [ 3.797626] sched_clock: Marking stable (3797547095, 0)->(4745007966, -947460871) [ 3.800959] registered taskstats version 1 [ 3.803150] Loading compiled-in X.509 certificates [ 3.805108] zswap: loaded using pool lzo/zbud [ 3.831700] Key type big_key registered [ 3.843127] Key type encrypted registered [ 3.844501] ima: No TPM chip found, activating TPM-bypass! [ 3.846110] ima: Allocated hash algorithm: sha1 [ 3.847398] ima: No architecture policies found [ 3.848757] evm: Initialising EVM extended attributes: [ 3.850295] evm: security.selinux [ 3.851353] evm: security.ima [ 3.852213] evm: security.capability [ 3.853273] evm: HMAC attrs: 0x1 [ 3.855672] rtc_cmos 00:05: setting system clock to 2026-09-03 17:06:49 UTC (1788455209) [ 3.861636] debug: unmapping init [mem 0xffffffff8a003000-0xffffffff8a1fffff] [ 3.864238] debug: unmapping init [mem 0xffffffff88d82000-0xffffffff89058fff] [ 3.872313] Write protecting the kernel read-only data: 28672k [ 3.875437] debug: unmapping init [mem 0xffffffff87403000-0xffffffff875fffff] [ 3.877716] debug: unmapping init [mem 0xffffffff87d14000-0xffffffff87dfffff] [ 3.912431] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.919618] systemd[1]: Detected virtualization kvm. [ 3.921324] systemd[1]: Detected architecture x86-64. [ 3.922821] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.949913] systemd[1]: No hostname configured. [ 3.951279] systemd[1]: Set hostname to . [ 3.952896] random: systemd: uninitialized urandom read (16 bytes read) [ 3.954754] systemd[1]: Initializing machine ID from random generator. [ 3.994655] random: ln: uninitialized urandom read (6 bytes read) [ 4.100263] random: systemd: uninitialized urandom read (16 bytes read) [ 4.103180] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.107927] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.112120] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.775927] device-mapper: uevent: version 1.0.3 [ 4.778801] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 5.506444] random: fast init done [ 5.522101] virtio_net virtio0 ens2: renamed from eth0 [ 5.600315] scsi host0: ata_piix [ 5.613709] scsi host1: ata_piix [ 5.626982] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.630607] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.109253] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.363300] random: crng init done [ 10.364900] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.822220] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.962449] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.257146] SELinux: Disabled at runtime. [ 12.320114] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.330117] systemd[1]: Detected virtualization kvm. [ 12.331996] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.928725] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.932352] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.937252] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.941312] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.943787] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.956452] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.963710] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ 13.022778] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.453209] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.734504] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.824095] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.945131] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.956048] EDAC sbridge: Ver: 1.1.2 [ 18.671984] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (5s / no limit) [** ] A start job is running for Configur…-only root support (6s / no limit)[ 19.202227] NFS: Registering the id_resolver key type [ 19.204163] Key type id_resolver registered [ 19.206478] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Login Service. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg421-server login: [ 83.455563] hrtimer: interrupt took 2883518 ns [ 92.527388] libcfs: loading out-of-tree module taints kernel. [ 92.588521] Key type ._llcrypt registered [ 92.591492] Key type .llcrypt registered [ 92.789382] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 114.556911] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 115.829183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 115.844706] alg: No test for adler32 (adler32-zlib) [ 117.306230] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 118.130547] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 119.952317] Key type lgssc registered [ 121.818052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.316710] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 189.800761] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 205.383215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 205.415526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.714961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.794185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 206.940217] Lustre: lustre-MDT0000: new disk, initializing [ 207.092145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.140845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.146920] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.504172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.641803] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 229.669027] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 229.673194] Lustre: Skipped 1 previous similar message [ 229.814325] Lustre: lustre-MDT0001: new disk, initializing [ 229.944845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 229.979986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.001434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 234.626529] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.361503] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 253.546760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.984687] Lustre: lustre-OST0000: new disk, initializing [ 253.998886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 254.004307] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.093144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.599263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.610139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.694622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 261.167925] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.340232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 277.523500] Lustre: lustre-OST0001: new disk, initializing [ 277.532817] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 277.539782] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 277.640981] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 282.726151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 282.742221] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 285.500177] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 288.256065] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 298.552438] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 310.836726] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 318.008976] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing check_logdir /tmp/testlogs/ [ 324.053617] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing yml_node [ 330.450421] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 334.466568] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 338.442212] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 341.186563] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 13:12:24 EDT 2026 [ 362.731908] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 364.536463] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 380.385799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.387159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 380.404790] Lustre: Skipped 1 previous similar message [ 385.507250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.514793] Lustre: Skipped 6 previous similar messages [ 386.361375] Lustre: server umount lustre-MDT0000 complete [ 390.244356] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455595 with bad export cookie 9173249791341907446 [ 390.246530] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 390.259685] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 390.626166] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 390.629208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 390.666875] Lustre: Skipped 3 previous similar messages [ 390.691531] Lustre: Skipped 1 previous similar message [ 395.744976] LustreError: 10099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.745163] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 395.803156] LustreError: 10099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 396.868124] Lustre: server umount lustre-MDT0001 complete [ 400.885246] Lustre: server umount lustre-OST0000 complete [ 405.384217] Lustre: server umount lustre-OST0001 complete [ 413.523771] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 426.720234] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 476.388073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.603543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 476.633043] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 476.708341] Lustre: lustre-MDT0000: new disk, initializing [ 476.772908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.800739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 481.959751] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 494.376271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.550826] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 494.632243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 494.639284] Lustre: Skipped 1 previous similar message [ 494.708731] Lustre: lustre-MDT0001: new disk, initializing [ 494.855991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 494.908901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 494.924708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 500.291676] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 511.139324] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 517.542271] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 524.980775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 525.377352] Lustre: lustre-OST0000: new disk, initializing [ 525.380612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 525.391497] Lustre: 17056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 525.557466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.147463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 527.165657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 527.290209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 532.763972] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 542.597361] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 547.525714] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 547.707543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 553.267330] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 553.524466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 558.052541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.072211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 558.073196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 558.088871] Lustre: Skipped 2 previous similar messages [ 561.974609] Lustre: server umount lustre-OST0000 complete [ 566.755690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 566.762550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.772496] Lustre: Skipped 1 previous similar message [ 566.777086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 580.063854] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 580.264808] Lustre: server umount lustre-MDT0000 complete [ 582.627524] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.648497] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 583.260036] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455788 with bad export cookie 9173249791341908979 [ 583.261503] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 583.272876] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 583.580742] Lustre: server umount lustre-MDT0001 complete [ 589.863125] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:16:33 (1788455793) [ 607.430252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.955303] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.036067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 612.091408] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 613.350195] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.378247] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 617.442512] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.995253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.550268] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 635.544815] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 642.275792] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 645.264151] Lustre: Failing over lustre-MDT0000 [ 645.932084] Lustre: server umount lustre-MDT0000 complete [ 646.119476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 646.135566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.150764] Lustre: Skipped 1 previous similar message [ 646.160566] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.548770] LustreError: 20657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.562077] LustreError: 20657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 665.567181] Lustre: 3641:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455855/real 1788455855] req@ffff9160ffaed180 x1875331328941568/t0(0) o400->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788455871 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 665.608869] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 666.810356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.227436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 676.233263] Lustre: Skipped 1 previous similar message [ 676.318493] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 681.278567] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 681.448567] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 681.455620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 681.481894] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 690.078597] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 694.757158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.762357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 694.767166] Lustre: Skipped 2 previous similar messages [ 694.775989] Lustre: Skipped 6 previous similar messages [ 700.158618] Lustre: server umount lustre-MDT0000 complete [ 703.900742] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455909 with bad export cookie 9173249791341910393 [ 703.907808] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.918167] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 704.218490] Lustre: server umount lustre-MDT0001 complete [ 712.513374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.017801] LustreError: 23272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.045426] LustreError: 23272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 717.531723] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 725.580919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.351536] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 740.487616] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 747.063562] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 749.350400] Lustre: Failing over lustre-MDT0000 [ 750.437475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.634455] Lustre: server umount lustre-MDT0000 complete [ 766.943715] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455956/real 1788455956] req@ffff915fc4056680 x1875331328970880/t0(0) o400->MGC192.168.204.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788455972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 767.019879] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 775.070802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.479188] LustreError: 25262:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 776.510072] LustreError: 25262:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9160ffaed880 x1875331328972800/t0(0) o250->MGC192.168.204.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788455982 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 776.568280] LustreError: 25262:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 777.379411] LustreError: 24002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 777.421756] LustreError: 24002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 778.098470] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 783.329346] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 783.335813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 783.358237] Lustre: Skipped 1 previous similar message [ 783.379583] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 784.432156] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 796.150571] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 800.736856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.761912] Lustre: Skipped 1 previous similar message [ 800.780799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 800.803228] Lustre: Skipped 4 previous similar messages [ 806.554495] Lustre: server umount lustre-MDT0000 complete [ 811.621538] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456017 with bad export cookie 9173249791341911331 [ 811.660407] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 812.067610] Lustre: server umount lustre-MDT0001 complete [ 822.765366] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:20:26 (1788456026) [ 841.850806] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 851.918631] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 901.625378] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 916.226981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.455190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 916.498600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 916.625282] Lustre: lustre-MDT0000: new disk, initializing [ 916.765540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.782050] Lustre: Skipped 3 previous similar messages [ 916.804760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 922.367783] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 933.328838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 933.451258] Lustre: 29646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 933.722869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 938.958364] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 950.134144] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 958.549916] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 966.011575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 966.426389] Lustre: 31615:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 968.482475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 968.556179] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 968.661218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 968.696557] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 973.905891] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 984.506503] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 991.219373] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1014.758058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.758244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1014.769881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1014.769891] Lustre: Skipped 2 previous similar messages [ 1014.771995] Lustre: Skipped 2 previous similar messages [ 1014.791880] LustreError: Skipped 1 previous similar message [ 1019.675577] Lustre: server umount lustre-OST0000 complete [ 1030.821754] Lustre: server umount lustre-MDT0000 complete [ 1035.021112] LustreError: 29638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456240 with bad export cookie 9173249791341911737 [ 1035.033220] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1035.055411] LustreError: 29638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1035.643307] Lustre: server umount lustre-MDT0001 complete [ 1050.579512] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1053.474253] Key type lgssc unregistered [ 1053.810271] LNet: 33601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.818802] LNetError: 33601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.886735] LNet: Removed LNI 192.168.204.121@tcp [ 1055.873275] Key type .llcrypt unregistered [ 1055.876263] Key type ._llcrypt unregistered [ 1084.292604] Key type ._llcrypt registered [ 1084.294994] Key type .llcrypt registered [ 1084.428428] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1086.134263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1086.347689] alg: No test for adler32 (adler32-zlib) [ 1087.419457] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1087.775764] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1089.552240] Key type lgssc registered [ 1091.432875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1101.506962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1101.532450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.282761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.585684] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1112.546183] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1112.566693] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1117.669785] LustreError: 35014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.273830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.955873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1125.288232] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1136.815053] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1145.291290] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1155.105909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1156.106475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.833588] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1165.303098] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1172.517401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1174.472600] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1192.503965] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:26:36 (1788456396) [ 1194.977139] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1197.216589] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1205.725234] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:26:49 (1788456409) [ 1217.071742] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:27:00 (1788456420) [ 1218.631206] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1229.904692] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1229.910604] Lustre: Skipped 2 previous similar messages [ 1231.894716] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:27:15 (1788456435) [ 1234.321932] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1234.330554] Lustre: Skipped 2 previous similar messages [ 1242.554830] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1242.562337] Lustre: Skipped 1 previous similar message [ 1259.064877] Lustre: Disabling parameter general.osc.*.max_pages_per_rpc= in log params [ 1259.075097] Lustre: Skipped 8 previous similar messages [ 1260.913324] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:27:44 (1788456464) [ 1264.260474] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1268.429769] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1283.192204] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:28:05 (1788456485) [ 1293.131607] Lustre: Disabling parameter general.jobid_name=TESTNAME1 in log params [ 1293.135778] Lustre: Skipped 3 previous similar messages [ 1307.427488] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:28:29 (1788456509) [ 1326.053976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1326.055963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.116100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1326.123958] Lustre: Skipped 1 previous similar message [ 1330.544264] Lustre: server umount lustre-OST0000 complete [ 1339.874466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1339.900938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.939646] Lustre: Skipped 1 previous similar message [ 1339.951795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1341.930573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.932934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1341.959720] Lustre: Skipped 1 previous similar message [ 1343.520059] Lustre: server umount lustre-MDT0000 complete [ 1346.084031] LustreError: 35014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.108667] LustreError: 35014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1348.863827] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456554 with bad export cookie 2968597200199265075 [ 1348.886458] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1348.887865] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1349.260863] Lustre: server umount lustre-MDT0001 complete [ 1363.000820] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1366.671343] Key type lgssc unregistered [ 1367.052814] LNet: 42883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.065738] LNetError: 42883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.086337] LNet: Removed LNI 192.168.204.121@tcp [ 1368.260293] Key type .llcrypt unregistered [ 1368.262709] Key type ._llcrypt unregistered [ 1391.135111] Key type ._llcrypt registered [ 1391.137820] Key type .llcrypt registered [ 1391.313711] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 1412.908629] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1413.919202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1414.041376] alg: No test for adler32 (adler32-zlib) [ 1415.589266] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1416.056977] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1417.831717] Key type lgssc registered [ 1419.859806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1470.505082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1470.548572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.859646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1471.899323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1472.018502] Lustre: lustre-MDT0000: new disk, initializing [ 1472.175085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.209359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1478.671289] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1496.149110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.500761] Lustre: 46037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1496.648705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1496.652131] Lustre: Skipped 1 previous similar message [ 1496.841758] Lustre: lustre-MDT0001: new disk, initializing [ 1497.034098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1497.125652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1497.150949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1504.281974] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1516.824831] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1523.477833] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1532.561636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1532.905520] Lustre: lustre-OST0000: new disk, initializing [ 1532.918934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1532.932546] Lustre: 48009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1533.035949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.507838] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1542.737938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1542.752602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1542.881097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1555.138134] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1562.780567] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1563.170510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1569.687352] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1570.013411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1573.860816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1573.867312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1573.892088] LustreError: Skipped 1 previous similar message [ 1573.896974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1573.960371] Lustre: Skipped 1 previous similar message [ 1579.376369] Lustre: server umount lustre-OST0000 complete [ 1589.216826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1589.219504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1589.225165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1589.248303] Lustre: Skipped 1 previous similar message [ 1589.268324] Lustre: Skipped 2 previous similar messages [ 1592.109126] Lustre: server umount lustre-MDT0000 complete [ 1594.343479] LustreError: 46999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1594.378116] LustreError: 46999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1597.388242] LustreError: 46029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456803 with bad export cookie 1508934157109102073 [ 1597.400582] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1597.414948] LustreError: 46029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1598.053646] Lustre: server umount lustre-MDT0001 complete [ 1612.004674] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:33:35 (1788456815) [ 1620.892502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.685789] LustreError: 50474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.884866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.122112] LustreError: 50475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1627.310824] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1632.237729] LustreError: 50474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1637.357712] LustreError: 50475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.013627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.670770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1644.259360] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1653.502050] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1661.634438] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1672.057315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1672.627553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.342875] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1689.343612] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1698.646162] LustreError: 53009:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1699.640550] Lustre: Modifying parameter general.timeout=1 in log params [ 1700.858237] Lustre: Modifying parameter general.timeout=2 in log params [ 1703.219676] Lustre: Modifying parameter general.timeout=4 in log params [ 1703.223432] Lustre: Skipped 1 previous similar message [ 1707.826934] Lustre: Modifying parameter general.timeout=8 in log params [ 1707.833125] Lustre: Skipped 3 previous similar messages [ 1716.646076] Lustre: Modifying parameter general.timeout=16 in log params [ 1716.648674] Lustre: Skipped 7 previous similar messages [ 1732.734434] Lustre: Modifying parameter general.timeout=30 in log params [ 1732.740113] Lustre: Skipped 13 previous similar messages [ 1765.701478] Lustre: Modifying parameter general.timeout=57 in log params [ 1765.707595] Lustre: Skipped 26 previous similar messages [ 1829.950921] Lustre: Modifying parameter general.timeout=117 in log params [ 1829.958188] Lustre: Skipped 59 previous similar messages [ 1857.709065] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:37:40 (1788457060) [ 1861.098247] Lustre: 59972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1862.988515] Lustre: 60070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1865.252390] Lustre: 60170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1869.724823] Lustre: 60379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1869.748432] Lustre: 60379:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1903.267075] Lustre: 61840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1903.273848] Lustre: 61840:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1903.315278] LustreError: 61840:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1908.541816] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:38:31 (1788457111) [ 1925.610114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1925.618162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.640897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.124680] Lustre: server umount lustre-MDT0000 complete [ 1931.244112] LustreError: 50474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.267206] LustreError: 50474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1933.501715] LustreError: 62456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457139 with bad export cookie 1508934157109102955 [ 1933.509497] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.519893] LustreError: 62456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1933.985447] Lustre: server umount lustre-MDT0001 complete [ 1949.407359] Lustre: server umount lustre-OST0000 complete [ 1977.328598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.607651] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1977.652804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1977.665374] Lustre: Skipped 35 previous similar messages [ 1977.829566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.090772] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1991.757413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.869564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1991.904757] Lustre: 64375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1997.264436] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2005.918205] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2012.416097] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2020.676950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2020.961602] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2021.346665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2021.351870] Lustre: Skipped 1 previous similar message [ 2028.159349] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2037.119416] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2039.279307] Lustre: 66945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2047.969263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2047.970809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.978815] LustreError: Skipped 1 previous similar message [ 2047.980104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2047.996909] Lustre: Skipped 3 previous similar messages [ 2048.018146] Lustre: Skipped 4 previous similar messages [ 2053.100140] LustreError: 66223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.127776] LustreError: 66223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2053.230403] Lustre: server umount lustre-OST0000 complete [ 2058.212288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.224793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.230375] Lustre: Skipped 1 previous similar message [ 2058.250766] Lustre: Skipped 1 previous similar message [ 2063.663945] Lustre: server umount lustre-MDT0000 complete [ 2068.505503] LustreError: 64366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457274 with bad export cookie 1508934157109106245 [ 2068.508035] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2068.519740] LustreError: 64366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2069.317830] Lustre: server umount lustre-MDT0001 complete [ 2081.991647] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2085.169279] Key type lgssc unregistered [ 2085.670348] LNet: 68157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.691059] LNetError: 68157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2086.762229] LNet: Removed LNI 192.168.204.121@tcp [ 2087.929144] Key type .llcrypt unregistered [ 2087.931327] Key type ._llcrypt unregistered [ 2112.688632] Key type ._llcrypt registered [ 2112.691573] Key type .llcrypt registered [ 2112.817296] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2113.457696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2113.596823] alg: No test for adler32 (adler32-zlib) [ 2114.814276] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2115.064387] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2116.847230] Key type lgssc registered [ 2118.370530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2131.730869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2131.761257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.595074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.209980] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2143.723849] LustreError: 69567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.744152] LustreError: 69567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2148.286769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.891822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2154.573710] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2163.511492] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2170.210056] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2177.510827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2178.046116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.433414] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2192.732481] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2202.098441] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2213.863752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.870276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.882970] Lustre: Skipped 2 previous similar messages [ 2213.900084] Lustre: Skipped 1 previous similar message [ 2217.616323] Lustre: server umount lustre-MDT0000 complete [ 2218.996965] LustreError: 69567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2219.017810] LustreError: 69567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2222.260702] LustreError: 70308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457427 with bad export cookie 6342958753838892206 [ 2222.278174] LustreError: 70308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2222.285054] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2223.166932] Lustre: server umount lustre-MDT0001 complete [ 2237.803145] Lustre: server umount lustre-OST0000 complete [ 2271.968424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.381315] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2272.436829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2272.796422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.451101] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2289.157731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.361392] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2289.406755] Lustre: 74443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2289.870627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2294.837622] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2303.974419] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2309.640727] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2317.031652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2317.211850] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2317.712605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.121934] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2334.459991] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2336.351647] Lustre: 77018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2338.775299] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2347.489628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2347.503439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.516949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2348.006677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2350.543361] Lustre: server umount lustre-OST0000 complete [ 2356.195589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2356.217653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2356.235693] Lustre: Skipped 1 previous similar message [ 2356.247317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2356.256420] Lustre: Skipped 1 previous similar message [ 2358.246408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.259249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2358.272852] Lustre: Skipped 1 previous similar message [ 2361.389936] Lustre: server umount lustre-MDT0000 complete [ 2363.365432] LustreError: 74453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.408931] LustreError: 74453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2366.551458] LustreError: 74435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457572 with bad export cookie 6342958753838893053 [ 2366.561571] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.582320] LustreError: 74435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2367.561502] Lustre: server umount lustre-MDT0001 complete [ 2380.715346] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2383.517086] Key type lgssc unregistered [ 2383.842663] LNet: 78227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2383.849028] LNetError: 78227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2383.870769] LNet: Removed LNI 192.168.204.121@tcp [ 2384.803542] Key type .llcrypt unregistered [ 2384.806261] Key type ._llcrypt unregistered [ 2412.503527] Key type ._llcrypt registered [ 2412.506260] Key type .llcrypt registered [ 2412.660216] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2413.808708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2414.053438] alg: No test for adler32 (adler32-zlib) [ 2415.186281] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2415.504670] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2417.239185] Key type lgssc registered [ 2418.711555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2426.737492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2426.760813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.198962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.270943] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2437.626465] LustreError: 79636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.656677] LustreError: 79636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2442.727404] LustreError: 79637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.532828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.505360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2450.370143] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2461.366968] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2469.323651] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2478.916326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2479.907981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.368212] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2499.164926] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2505.817856] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2507.726334] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2509.263326] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2511.989498] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2511.993616] Lustre: Skipped 1 previous similar message [ 2516.327182] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2516.333884] Lustre: Skipped 2 previous similar messages [ 2524.590895] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2524.599646] Lustre: Skipped 5 previous similar messages [ 2541.478362] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2541.486823] Lustre: Skipped 13 previous similar messages [ 2574.178953] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2574.187414] Lustre: Skipped 31 previous similar messages [ 2642.324928] Lustre: 87761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2643.554047] Lustre: Disabling parameter general.jobid_var= in log params [ 2643.560821] Lustre: Skipped 40 previous similar messages [ 2646.373926] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:50:49 (1788457849) [ 2648.329230] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2650.591170] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:50:54 (1788457854) [ 2658.276902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.278902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.289186] LustreError: Skipped 1 previous similar message [ 2658.313847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2660.535429] Lustre: server umount lustre-OST0000 complete [ 2668.521418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.527637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2668.545270] Lustre: Skipped 2 previous similar messages [ 2668.555721] Lustre: Skipped 2 previous similar messages [ 2671.799635] Lustre: server umount lustre-MDT0000 complete [ 2673.643395] LustreError: 80368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.677797] LustreError: 80368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2675.914575] LustreError: 79616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457881 with bad export cookie 13642680397708518192 [ 2675.916753] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2675.925188] LustreError: 79616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2676.405643] Lustre: server umount lustre-MDT0001 complete [ 2688.793502] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2691.967806] Key type lgssc unregistered [ 2692.356716] LNet: 89011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2692.364987] LNetError: 89011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2693.413850] LNet: Removed LNI 192.168.204.121@tcp [ 2694.359420] Key type .llcrypt unregistered [ 2694.362938] Key type ._llcrypt unregistered [ 2699.606416] Key type ._llcrypt registered [ 2699.608345] Key type .llcrypt registered [ 2699.711366] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2704.237486] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules [ 2705.495787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2705.609659] alg: No test for adler32 (adler32-zlib) [ 2706.837829] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2707.045190] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2708.745427] Key type lgssc registered [ 2709.971116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2711.899479] LustreError: 90210:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2734.138869] LustreError: 90210:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2735.286961] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2746.680711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2746.712154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.142299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.408320] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2757.603730] LustreError: 91232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2757.622561] LustreError: 91232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2762.720249] LustreError: 91233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.939303] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:52:49 (1788457969) [ 2767.842842] LustreError: 91232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.905427] LustreError: 91264:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2771.922117] LustreError: 91264:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2771.932323] LustreError: 91264:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 2772.490391] Lustre: server umount lustre-MDT0000 complete [ 2788.342969] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2791.398275] Key type lgssc unregistered [ 2791.747530] LNet: 92757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2791.758658] LNetError: 92757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2791.777950] LNet: Removed LNI 192.168.204.121@tcp [ 2792.880715] Key type .llcrypt unregistered [ 2792.888882] Key type ._llcrypt unregistered [ 2813.632586] Key type ._llcrypt registered [ 2813.643952] Key type .llcrypt registered [ 2813.838839] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2815.621626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2815.738185] alg: No test for adler32 (adler32-zlib) [ 2816.889574] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2817.380841] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2819.239335] Key type lgssc registered [ 2820.358813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2831.423629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2831.460466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.080068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.675893] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2842.598107] LustreError: 93974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.678791] LustreError: 93974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2847.716371] LustreError: 93973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2849.637578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.623902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2856.312917] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2865.058764] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2871.686321] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2879.616648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2880.421844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2886.843868] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2894.716452] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2957.793279] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:56:01 (1788458161) [ 2960.044661] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2962.307681] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:56:05 (1788458165) [ 2968.045453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2968.059488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.092364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.594472] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.594703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.618404] Lustre: Skipped 2 previous similar messages [ 2973.082944] Lustre: server umount lustre-MDT0000 complete [ 2975.719633] LustreError: 93973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.743937] LustreError: 93973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2977.286396] LustreError: 93954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788458182 with bad export cookie 8812322809495294276 [ 2977.310120] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2977.748759] Lustre: server umount lustre-MDT0001 complete [ 2993.155218] Lustre: server umount lustre-OST0000 complete [ 3005.122766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.609978] LustreError: 98290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.810211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.757857] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3011.044394] LustreError: 98291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.893996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3021.447168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3026.639418] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3035.387769] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3042.652304] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3057.237750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3057.476444] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3057.487796] LustreError: 98277:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3057.508770] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3057.519578] LustreError: 100375:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 3057.526353] LustreError: 100375:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 3057.532236] LustreError: 100375:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3057.592403] Lustre: server umount lustre-OST0000 complete [ 3057.595364] LustreError: 100375:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3067.007277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3067.215738] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3067.226289] LustreError: 98280:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3067.264719] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3067.272707] LustreError: 100842:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 3067.280617] LustreError: 100842:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 3067.289270] LustreError: 100842:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3067.378458] Lustre: server umount lustre-OST0000 complete [ 3067.381475] LustreError: 100842:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3077.184322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3077.446783] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3077.451986] Lustre: Found index 0 for lustre-OST0000, updating log [ 3077.457958] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3077.530293] Lustre: lustre-OST0000: new disk, initializing [ 3077.536401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3077.543199] Lustre: 101378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3077.953780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.612803] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3089.052665] Lustre: Failing over lustre-OST0000 [ 3089.352458] Lustre: server umount lustre-OST0000 complete [ 3097.568737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.573900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.582964] Lustre: Skipped 2 previous similar messages [ 3097.597135] Lustre: Skipped 1 previous similar message [ 3100.424743] Lustre: server umount lustre-MDT0000 complete [ 3102.693717] LustreError: 98290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3102.721573] LustreError: 98290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3104.523229] LustreError: 100394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788458310 with bad export cookie 8812322809495297020 [ 3104.532621] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3104.540528] LustreError: 100394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3104.994537] Lustre: server umount lustre-MDT0001 complete [ 3117.248726] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:58:39 (1788458319) [ 3139.325163] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3151.470471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3152.145490] LustreError: 103750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.379137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3158.721993] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3169.478728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.496109] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3179.834492] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3189.544637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3190.470942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.476029] Lustre: Skipped 1 previous similar message [ 3198.176784] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3208.383889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3208.555152] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3208.643337] Lustre: lustre-OST0001: new disk, initializing [ 3208.647761] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3208.655461] Lustre: 106067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3214.970154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3214.977758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3215.096666] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3216.421969] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3225.372409] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3237.390101] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3245.117721] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (stopping) [ 3246.054237] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.076990] Lustre: Skipped 1 previous similar message [ 3248.585481] Lustre: server umount lustre-MDT0001 complete [ 3250.207636] LustreError: 103745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.229983] LustreError: 103745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3259.661489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.820455] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3259.830182] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3259.898961] Lustre: 103740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3260.809720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3260.830766] Lustre: Skipped 1 previous similar message [ 3261.181189] LustreError: 107607:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3261.197035] LustreError: 107607:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3261.215398] LustreError: 103743:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3261.345567] LustreError: 107623:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.121@tcp: cfg command failed: rc = -17 [ 3261.371180] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3261.371180] [ 3266.026831] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3266.400602] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3277.997992] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 14:01:21 (1788458481) [ 3300.407208] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3319.575433] Lustre: 109720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3343.073787] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3347.998132] Lustre: 110854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3439.584144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3439.596374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.612336] Lustre: Skipped 1 previous similar message [ 3439.617533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.623605] Lustre: Skipped 3 previous similar messages [ 3441.933639] Lustre: server umount lustre-MDT0000 complete [ 3445.217276] LustreError: 103746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3445.243544] LustreError: 103746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3447.709846] LustreError: 103733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788458653 with bad export cookie 8812322809495298014 [ 3447.719669] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3447.740254] LustreError: 103733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3448.318210] Lustre: server umount lustre-MDT0001 complete [ 3464.220629] Lustre: server umount lustre-OST0000 complete [ 3466.207225] Lustre: 93283:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788458655/real 1788458655] req@ffff915fcc6b4380 x1875334159254912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788458671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3466.250177] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.266131] Lustre: Skipped 3 previous similar messages [ 3468.930638] Lustre: server umount lustre-OST0001 complete [ 3475.831825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3487.226338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3495.990486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3506.887336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3530.153765] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3539.788543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.806742] Lustre: lustre-MDT0000: reset Object Index mappings [ 3540.246379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.053610] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3554.463065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.485862] Lustre: lustre-MDT0001: reset Object Index mappings [ 3559.883467] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3563.154671] Lustre: 115955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3570.767984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3578.736199] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3579.823907] LustreError: 116309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.856284] LustreError: 116309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3579.856929] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3585.036258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3588.439284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3594.243630] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3594.244325] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3596.949124] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3608.001798] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3613.149582] Lustre: 117833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3667.862523] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 14:07:51 (1788458871) [ 3696.620502] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.632847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3696.642751] Lustre: Skipped 5 previous similar messages [ 3701.220266] Lustre: server umount lustre-MDT0000 complete [ 3705.242444] LustreError: 114785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788458910 with bad export cookie 8812322809495344865 [ 3705.261772] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3705.270737] LustreError: 114785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3711.793992] Lustre: server umount lustre-MDT0001 complete [ 3721.891050] Lustre: server umount lustre-OST0000 complete [ 3726.202976] LustreError: 118797:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0xb2:0x0].0x0 (ffff915fc37b7f00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3732.500795] Lustre: server umount lustre-OST0001 complete [ 3739.747663] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 3751.970956] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3791.251939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3791.494858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3791.578819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3791.709782] Lustre: lustre-MDT0000: new disk, initializing [ 3791.800219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3795.857944] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3803.533672] Lustre: server umount lustre-MDT0000 complete [ 3814.953677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.439167] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3815.453952] Lustre: 122111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3815.467909] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3815.726659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3815.734401] Lustre: Skipped 4 previous similar messages [ 3822.027554] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3827.988903] Lustre: server umount lustre-MDT0000 complete [ 3839.175845] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 14:10:42 (1788459042) [ 3841.068974] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3843.353466] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 14:10:46 (1788459046) [ 3865.993256] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3878.684425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3884.518730] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3894.848087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3894.962843] Lustre: 124160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3895.019782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3895.023899] Lustre: Skipped 1 previous similar message [ 3895.101834] Lustre: lustre-MDT0001: new disk, initializing [ 3895.313349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3895.347078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3900.505604] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3905.923427] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3913.443217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3913.789985] Lustre: lustre-OST0000: new disk, initializing [ 3913.795884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3913.802125] Lustre: 125740:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3915.180718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3915.198500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3915.428157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3921.221609] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3933.444962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3933.726828] Lustre: lustre-OST0001: new disk, initializing [ 3933.739418] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3933.753709] Lustre: 126611:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3935.748319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3935.763182] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3935.908597] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3941.751705] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3953.148771] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3957.450965] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4362.875815] Lustre: 133029:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4362.884931] Lustre: 133029:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4362.892917] Lustre: 133029:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4362.904219] Lustre: 133029:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4362.913822] Lustre: 133029:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4362.938148] Lustre: 133029:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4691.480630] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:24:53 (1788459893) [ 4709.373819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4709.379207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4709.392288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4709.401961] Lustre: Skipped 8 previous similar messages [ 4709.414059] Lustre: Skipped 9 previous similar messages [ 4714.199719] Lustre: server umount lustre-MDT0000 complete [ 4714.476347] LustreError: 127430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4714.511410] LustreError: 127430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 4718.530341] LustreError: 125369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459924 with bad export cookie 8812322809495414501 [ 4718.538922] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4718.548134] LustreError: 125369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4719.585090] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.600183] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4719.600922] Lustre: Skipped 1 previous similar message [ 4719.618604] Lustre: Skipped 1 previous similar message [ 4723.634255] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4723.649258] Lustre: Skipped 1 previous similar message [ 4725.236773] Lustre: server umount lustre-MDT0001 complete [ 4735.845894] Lustre: server umount lustre-OST0000 complete [ 4747.269580] Lustre: server umount lustre-OST0001 complete [ 4755.590447] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 4771.831935] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 4818.171160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4818.454937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4818.482831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4818.569620] Lustre: lustre-MDT0000: new disk, initializing [ 4818.639860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4818.650173] Lustre: Skipped 4 previous similar messages [ 4818.687849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4823.759778] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4835.491034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.643730] Lustre: 136119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4835.753893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4835.759443] Lustre: Skipped 1 previous similar message [ 4835.907427] Lustre: lustre-MDT0001: new disk, initializing [ 4836.110466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4836.137019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4841.320828] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4851.901837] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4857.880412] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4864.816393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4865.177978] Lustre: lustre-OST0000: new disk, initializing [ 4865.185374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4865.201444] Lustre: 138085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4866.729206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4866.742738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4866.855220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4872.586431] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 4883.090885] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4895.268964] Lustre: lustre-MDD0000: changelog on [ 4898.898709] Lustre: lustre-MDD0001: changelog on [ 7036.039827] Lustre: lustre-MDD0001: changelog off [ 7038.201260] Lustre: 137367:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7039.286958] Lustre: lustre-MDD0000: changelog off [ 7048.160407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7048.162286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.165607] LustreError: Skipped 1 previous similar message [ 7048.167090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7048.220266] Lustre: Skipped 1 previous similar message [ 7051.252665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7051.261355] Lustre: Skipped 2 previous similar messages [ 7052.798703] Lustre: server umount lustre-OST0000 complete [ 7058.407032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7058.421673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.440723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7061.478206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.732840] Lustre: server umount lustre-MDT0000 complete [ 7066.593923] LustreError: 136131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7066.613194] LustreError: 136131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7067.915695] LustreError: 136111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788462273 with bad export cookie 8812322809496237736 [ 7067.917583] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7067.939182] LustreError: 136111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7068.332955] Lustre: server umount lustre-MDT0001 complete [ 7080.819757] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 7083.742305] Key type lgssc unregistered [ 7084.076152] LNet: 147280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7084.094674] LNetError: 147280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7084.119704] LNet: Removed LNI 192.168.204.121@tcp [ 7085.034355] Key type .llcrypt unregistered [ 7085.039066] Key type ._llcrypt unregistered [ 7105.257212] Key type ._llcrypt registered [ 7105.258791] Key type .llcrypt registered [ 7105.360781] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 7121.398457] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 7122.556071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7122.607066] alg: No test for adler32 (adler32-zlib) [ 7123.702891] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7123.916937] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 7125.623181] Key type lgssc registered [ 7127.033278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7163.441436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7163.456888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7164.760330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7164.835791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7164.940316] Lustre: lustre-MDT0000: new disk, initializing [ 7164.987533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.005414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7169.023736] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7179.902035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.086513] Lustre: 150480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7180.144880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7180.148199] Lustre: Skipped 1 previous similar message [ 7180.243758] Lustre: lustre-MDT0001: new disk, initializing [ 7180.344186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7180.389704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7180.404211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7185.540731] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7195.446102] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7202.878844] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7211.030118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7211.412386] Lustre: lustre-OST0000: new disk, initializing [ 7211.423725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7211.430760] Lustre: 152448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7211.525080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7217.192227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7217.215909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7217.339194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7218.423747] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7227.926433] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7235.498063] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_module obdecho/obdecho [ 7238.164204] LustreError: 153451:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7238.171935] LustreError: 153451:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7240.243880] LustreError: 153549:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7257.061831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7257.068671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7257.080545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7258.595569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7258.597730] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7258.603647] Lustre: Skipped 1 previous similar message [ 7258.615453] Lustre: Skipped 1 previous similar message [ 7261.376643] Lustre: server umount lustre-MDT0000 complete [ 7265.099482] LustreError: 150471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788462470 with bad export cookie 15404153212309222073 [ 7265.111455] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7265.112496] LustreError: 150471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7265.476761] Lustre: server umount lustre-MDT0001 complete [ 7280.181136] Lustre: server umount lustre-OST0000 complete [ 7301.493954] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 7313.882170] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 7349.428943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7349.653878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7349.676250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7349.772126] Lustre: lustre-MDT0000: new disk, initializing [ 7349.871193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7349.887214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7354.122965] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7365.080839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.228226] Lustre: 157262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7365.279715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7365.283511] Lustre: Skipped 1 previous similar message [ 7365.384996] Lustre: lustre-MDT0001: new disk, initializing [ 7365.525065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7365.563364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7365.576693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7370.126754] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7380.695792] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7386.546759] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7393.868794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7394.233338] Lustre: lustre-OST0000: new disk, initializing [ 7394.244584] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7394.259987] Lustre: 159229:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7394.362349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7400.450309] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7402.008736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7402.018098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7402.153748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7410.061569] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7414.756166] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7414.937302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7419.730638] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7419.954592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7422.952945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7422.957134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7422.960478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7422.960487] Lustre: Skipped 1 previous similar message [ 7422.969181] LustreError: Skipped 1 previous similar message [ 7422.982028] Lustre: Skipped 1 previous similar message [ 7428.065282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7428.078046] Lustre: Skipped 1 previous similar message [ 7433.187314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7433.205645] Lustre: Skipped 2 previous similar messages [ 7436.255576] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7436.528376] Lustre: server umount lustre-OST0000 complete [ 7442.400699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7442.408516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.454212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7448.372684] Lustre: server umount lustre-MDT0000 complete [ 7448.549387] LustreError: 159883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7448.604805] LustreError: 159883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7453.626335] LustreError: 157255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788462659 with bad export cookie 15404153212309223032 [ 7453.628109] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7453.636090] LustreError: 157255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7454.103080] Lustre: server umount lustre-MDT0001 complete [ 7458.195941] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:11:01 (1788462661) [ 7460.133267] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7461.993585] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:11:05 (1788462665) [ 7470.452567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.910121] LustreError: 161648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7470.982595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7475.495687] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7476.196172] LustreError: 161649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7481.313716] LustreError: 161648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7483.729909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7487.861899] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7495.606242] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7501.718050] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7509.964995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7510.355750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7510.360551] Lustre: Skipped 1 previous similar message [ 7518.137704] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7526.569856] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7538.375806] Lustre: server umount lustre-MDT0000 complete [ 7539.170478] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.176838] LustreError: 164143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7539.193596] Lustre: Skipped 2 previous similar messages [ 7542.447171] LustreError: 161630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788462748 with bad export cookie 15404153212309223914 [ 7542.449027] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7542.460025] LustreError: 161630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7542.820684] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (stopping) [ 7542.841110] Lustre: Skipped 2 previous similar messages [ 7547.296135] LustreError: 161643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.311465] LustreError: 161643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7548.889397] Lustre: server umount lustre-MDT0001 complete [ 7555.306903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7568.283552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7568.571151] LustreError: 165489:0:(llog.c:1655:llog_backup()) MGC192.168.204.121@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7568.582628] Lustre: 165489:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.121@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7572.961436] LustreError: 165494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7572.970871] LustreError: 165494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7608.804127] LustreError: 165495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7608.831046] LustreError: 165495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 7629.279734] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7629.287632] Lustre: 165489:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.121@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7670.264638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7674.905205] LustreError: 165516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7674.923143] LustreError: 165516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 7675.025803] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7675.395293] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7684.258558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7687.651296] LustreError: 148698:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e9817586 x1875338675228672/t0(0) o250->MGC192.168.204.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7688.014815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 7692.825116] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7693.312125] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7701.084311] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7702.920762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7709.754935] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7711.531891] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7737.317832] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7737.333304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7737.348422] Lustre: Skipped 3 previous similar messages [ 7737.350049] Lustre: Skipped 3 previous similar messages [ 7740.379105] Lustre: server umount lustre-MDT0000 complete [ 7744.472065] LustreError: 161628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788462950 with bad export cookie 15404153212309230368 [ 7744.479495] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7744.488677] LustreError: 161628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7745.238484] Lustre: server umount lustre-MDT0001 complete [ 7760.709943] Lustre: server umount lustre-OST0000 complete [ 7797.464414] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 7813.976081] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 7861.417421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7861.739475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7861.788428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7861.880932] Lustre: lustre-MDT0000: new disk, initializing [ 7861.987775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7861.995347] Lustre: Skipped 1 previous similar message [ 7862.015592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7866.894560] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7878.517428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.591369] Lustre: 170622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7878.830566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7883.274894] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7893.696639] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7900.403345] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7908.167638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7908.476448] Lustre: lustre-OST0000: new disk, initializing [ 7908.479681] Lustre: Skipped 1 previous similar message [ 7908.493307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7908.500070] Lustre: Skipped 2 previous similar messages [ 7908.507536] Lustre: 172590:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7914.577069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7914.586073] Lustre: Skipped 1 previous similar message [ 7914.591203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7914.695752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7915.861627] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7925.991241] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7931.891819] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7932.141986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7938.010718] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7938.337446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7945.701430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7945.702403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7945.708490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7945.708504] Lustre: Skipped 2 previous similar messages [ 7945.713347] LustreError: Skipped 1 previous similar message [ 7945.755756] Lustre: Skipped 2 previous similar messages [ 7946.993629] Lustre: server umount lustre-OST0000 complete [ 7955.946951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7959.472949] Lustre: server umount lustre-MDT0000 complete [ 7961.061432] LustreError: 170627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7961.080324] LustreError: 170627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 7963.590020] LustreError: 170612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463169 with bad export cookie 15404153212309247203 [ 7963.600403] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7963.602545] LustreError: 170612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7964.082041] Lustre: server umount lustre-MDT0001 complete [ 7974.591141] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:19:38 (1788463178) [ 7982.109540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.946885] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 7998.344744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.546605] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8013.375258] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8020.911408] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8029.175032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8036.987271] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8045.486084] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8068.304886] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:21:11 (1788463271) [ 8073.699727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8073.700527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8073.713689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8073.726721] Lustre: Skipped 3 previous similar messages [ 8073.738409] Lustre: Skipped 4 previous similar messages [ 8079.601865] Lustre: server umount lustre-OST0000 complete [ 8090.461480] Lustre: server umount lustre-MDT0000 complete [ 8094.756737] LustreError: 174984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463300 with bad export cookie 15404153212309248085 [ 8094.760317] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8094.788594] LustreError: 174984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8095.200076] Lustre: server umount lustre-MDT0001 complete [ 8108.832971] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 8111.846664] Key type lgssc unregistered [ 8112.130683] LNet: 178757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8112.145186] LNetError: 178757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8112.162618] LNet: Removed LNI 192.168.204.121@tcp [ 8113.008611] Key type .llcrypt unregistered [ 8113.011675] Key type ._llcrypt unregistered [ 8137.263360] Key type ._llcrypt registered [ 8137.265516] Key type .llcrypt registered [ 8137.359219] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8138.571365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8138.622165] alg: No test for adler32 (adler32-zlib) [ 8139.747680] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8139.993612] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 8141.679177] Key type lgssc registered [ 8142.935969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8151.970392] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8152.005161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8167.712844] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8172.831339] LustreError: 179997:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 8177.953950] LustreError: 179997:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8177.990159] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8178.019391] LustreError: MGC192.168.204.121@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8178.058128] LustreError: 179997:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8178.081151] LustreError: 179997:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 8178.099340] LustreError: 179997:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8178.155404] Lustre: server umount lustre-OST0000 complete [ 8178.157613] LustreError: 179997:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8189.099828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8190.527470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8195.872837] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8200.674967] LustreError: 180497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8200.701116] LustreError: 180497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8205.798387] LustreError: 180498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8207.181705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8207.616539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8213.461722] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8223.478618] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8230.878032] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8239.250841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8239.579727] LustreError: 182316:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8239.601741] Lustre: 182316:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.121@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8239.811452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8247.433965] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8256.832658] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8263.651261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8263.654381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8263.680627] Lustre: Skipped 2 previous similar messages [ 8263.697646] Lustre: Skipped 2 previous similar messages [ 8268.774838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8268.784775] Lustre: Skipped 2 previous similar messages [ 8273.890704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8273.910316] Lustre: Skipped 2 previous similar messages [ 8274.400685] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8274.526734] Lustre: server umount lustre-MDT0000 complete [ 8279.011899] LustreError: 181634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.044776] LustreError: 181634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8279.294569] LustreError: 180478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463484 with bad export cookie 6757472986224050463 [ 8279.302899] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8279.311913] LustreError: 180478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8279.751469] Lustre: server umount lustre-MDT0001 complete [ 8293.701896] Lustre: server umount lustre-OST0000 complete [ 8303.521527] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 8317.066322] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8364.854344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.209444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8365.245235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8365.328083] Lustre: lustre-MDT0000: new disk, initializing [ 8365.421440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8365.451781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8370.756894] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8383.711337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8383.876676] Lustre: 185844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8383.939820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8383.946290] Lustre: Skipped 1 previous similar message [ 8384.034129] Lustre: lustre-MDT0001: new disk, initializing [ 8384.190696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8384.239643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8384.256740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8389.276955] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8402.188405] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8409.306217] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8419.481380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8419.850096] Lustre: lustre-OST0000: new disk, initializing [ 8419.857486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8419.866151] Lustre: 187815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8419.975070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8427.484769] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8428.130538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8428.143586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8428.259808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8439.056434] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8446.181732] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8446.505231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8452.305992] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8452.475643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8459.235801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8459.236329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8459.248387] LustreError: Skipped 1 previous similar message [ 8459.252062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8459.268668] Lustre: Skipped 1 previous similar message [ 8461.037773] Lustre: server umount lustre-OST0000 complete [ 8466.399990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8466.411114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8466.443919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8466.450831] Lustre: Skipped 1 previous similar message [ 8469.476586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8474.593482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8474.605023] Lustre: Skipped 3 previous similar messages [ 8479.201189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8479.475942] Lustre: server umount lustre-MDT0000 complete [ 8483.647624] LustreError: 185837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463689 with bad export cookie 6757472986224051177 [ 8483.654317] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8483.677312] LustreError: 185837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8484.040912] Lustre: server umount lustre-MDT0001 complete [ 8496.050125] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:28:19 (1788463699) [ 8519.208609] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 8522.350872] Key type lgssc unregistered [ 8522.740750] LNet: 190764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8522.755299] LNetError: 190764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8522.789789] LNet: Removed LNI 192.168.204.121@tcp [ 8523.994052] Key type .llcrypt unregistered [ 8523.995617] Key type ._llcrypt unregistered [ 8547.204948] Key type ._llcrypt registered [ 8547.206793] Key type .llcrypt registered [ 8547.295702] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8548.420409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8548.465282] alg: No test for adler32 (adler32-zlib) [ 8549.660701] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8549.895544] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 8551.535505] Key type lgssc registered [ 8552.864467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8562.002399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8562.018478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8563.441275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8569.992533] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8573.921913] LustreError: 191979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8573.934726] LustreError: 191979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8579.043729] LustreError: 191980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8581.601343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8582.162941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8589.501973] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8601.381414] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8609.383985] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8617.733947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8617.917413] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8617.921130] LustreError: 193845:0:(llog.c:1696:llog_backup()) MGC192.168.204.121@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8617.928544] Lustre: 193845:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.121@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8618.027314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8625.098725] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8634.422901] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8638.433742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8638.448798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8638.482243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8643.557870] LustreError: 192891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8643.568882] LustreError: 192891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8643.654547] Lustre: server umount lustre-MDT0000 complete [ 8648.020266] LustreError: 191959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788463853 with bad export cookie 6391041782465984279 [ 8648.027029] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8648.031852] LustreError: 191959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8648.466587] Lustre: server umount lustre-MDT0001 complete [ 8663.789337] Lustre: server umount lustre-OST0000 complete [ 8674.670247] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 8688.983200] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8734.412373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8734.775868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8734.827902] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8734.938891] Lustre: lustre-MDT0000: new disk, initializing [ 8735.061101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8735.081405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8740.541324] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8753.784452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8753.991356] Lustre: 197395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8754.047566] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8754.052226] Lustre: Skipped 1 previous similar message [ 8754.055675] LustreError: 198178:0:(llog.c:1696:llog_backup()) MGC192.168.204.121@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8754.067393] LustreError: 198178:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8754.074416] Lustre: 198178:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.121@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8754.099172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8754.103462] Lustre: Skipped 1 previous similar message [ 8754.190829] Lustre: lustre-MDT0001: new disk, initializing [ 8754.268248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8754.296046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8754.305987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8759.694295] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8769.182594] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8775.678569] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8783.111037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8783.554466] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8783.560104] Lustre: Skipped 1 previous similar message [ 8783.564375] LustreError: 199341:0:(llog.c:1696:llog_backup()) MGC192.168.204.121@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8783.570334] LustreError: 199341:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8783.575551] Lustre: 199341:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.121@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8783.583586] Lustre: 199341:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8783.614326] Lustre: lustre-OST0000: new disk, initializing [ 8783.618480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8783.624265] Lustre: 199364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8783.702723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8791.633700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8791.657649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8791.863678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8791.898150] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8802.735472] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8808.300717] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8808.467843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8813.747434] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8814.008300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8817.638110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8817.641983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8817.667814] Lustre: Skipped 3 previous similar messages [ 8817.685249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8817.693514] Lustre: Skipped 3 previous similar messages [ 8822.183786] Lustre: server umount lustre-OST0000 complete [ 8827.872717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8827.898425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8827.908465] Lustre: Skipped 1 previous similar message [ 8832.670645] Lustre: server umount lustre-MDT0000 complete [ 8832.995529] LustreError: 197402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8833.009784] LustreError: 197402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8836.710228] LustreError: 197388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464042 with bad export cookie 6391041782465985000 [ 8836.716063] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8836.720494] LustreError: 197388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8837.108962] Lustre: server umount lustre-MDT0001 complete [ 8848.674956] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:34:11 (1788464051) [ 8871.843445] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 8883.819915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.225554] LustreError: 202516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8884.294303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8889.217893] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8893.408368] LustreError: 202517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8893.456118] LustreError: 202517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8899.767674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8906.833452] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8910.422635] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8917.511709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8917.817957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8917.823680] Lustre: Skipped 1 previous similar message [ 8924.367066] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8933.358689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8933.589266] Lustre: lustre-OST0001: new disk, initializing [ 8933.593686] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8933.601502] Lustre: 204822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8940.350684] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8941.174971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8941.192284] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8941.320222] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8952.063982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8961.001856] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8963.278721] Lustre: DEBUG MARKER: ADD OST3 [ 8969.542763] loop: module loaded [ 8969.651333] loop0: detected capacity change from 0 to 204800000 [ 8969.841170] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8969.864416] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8969.896784] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8977.480483] Lustre: DEBUG MARKER: STOP OST3 [ 8977.529536] loop0: detected capacity change from 0 to 204800000 [ 8983.968827] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8984.218528] Lustre: lustre-OST0003: new disk, initializing [ 8984.237162] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8984.253459] Lustre: 206858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8984.462336] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8984.472774] Lustre: Skipped 1 previous similar message [ 8985.661080] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8985.814727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8985.821401] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8985.874342] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8987.680725] LustreError: 206980:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8987.913055] Lustre: Failing over lustre-OST0003 [ 8988.128918] Lustre: server umount lustre-OST0003 complete [ 8990.808211] LustreError: 204830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8990.835809] LustreError: 204830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8991.219906] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8991.220220] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8991.260338] Lustre: Skipped 1 previous similar message [ 8991.464128] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 8999.852641] Lustre: DEBUG MARKER: START OST3 again [ 9007.838549] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9008.276928] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 9009.907533] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9010.240090] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9010.257611] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9015.177521] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9038.517861] Lustre: Failing over lustre-OST0003 [ 9038.692529] Lustre: server umount lustre-OST0003 complete [ 9038.828160] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9038.841857] LustreError: 204830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9038.845868] Lustre: Skipped 2 previous similar messages [ 9038.860570] LustreError: 204830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 9059.297698] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.322738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9059.325547] Lustre: Skipped 1 previous similar message [ 9059.326512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9059.356748] Lustre: Skipped 3 previous similar messages [ 9064.516361] Lustre: server umount lustre-MDT0000 complete [ 9069.322773] LustreError: 204022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464274 with bad export cookie 6391041782465985882 [ 9069.325188] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9069.331705] LustreError: 204022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9069.550462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9069.568839] Lustre: Skipped 1 previous similar message [ 9075.765305] Lustre: server umount lustre-MDT0001 complete [ 9080.435052] Lustre: server umount lustre-OST0000 complete [ 9085.802751] Lustre: server umount lustre-OST0001 complete [ 9093.773619] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 9104.074450] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9147.066909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9147.334247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9147.407729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9147.498337] Lustre: lustre-MDT0000: new disk, initializing [ 9147.587049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9147.590551] Lustre: Skipped 1 previous similar message [ 9147.611138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9151.721227] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9162.179973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9162.267383] Lustre: 212284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9162.587447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9167.108711] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9176.958292] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9183.561598] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9191.055461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.342270] Lustre: lustre-OST0000: new disk, initializing [ 9191.346185] Lustre: Skipped 1 previous similar message [ 9191.352888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9191.363638] Lustre: Skipped 2 previous similar messages [ 9191.372473] Lustre: 214254:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9193.122926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9193.140195] Lustre: Skipped 1 previous similar message [ 9193.146131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9193.233941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9198.405625] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9209.010517] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9213.517643] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9213.760931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9218.708271] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9218.979134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9224.166397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9224.178374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9224.211633] Lustre: Skipped 4 previous similar messages [ 9224.223856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9224.238346] Lustre: Skipped 3 previous similar messages [ 9227.520962] Lustre: server umount lustre-OST0000 complete [ 9234.405555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9234.424649] LustreError: Skipped 1 previous similar message [ 9238.659631] Lustre: server umount lustre-MDT0000 complete [ 9239.532184] LustreError: 212297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9239.553199] LustreError: 212297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 9243.262175] LustreError: 212277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464448 with bad export cookie 6391041782465988178 [ 9243.265771] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9243.281597] LustreError: 212277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9243.869799] Lustre: server umount lustre-MDT0001 complete [ 9247.986819] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 15:40:51 (1788464451) [ 9268.017250] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 9280.701542] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9320.213876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9320.446946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9320.495504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9320.561983] Lustre: lustre-MDT0000: new disk, initializing [ 9320.673572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9324.869915] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9336.192327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9336.367868] Lustre: 218868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9336.712491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9340.595792] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9350.188843] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9356.272824] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9365.020742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9365.680347] Lustre: 220837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9368.061629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9375.639282] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9387.577139] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9392.317782] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9392.515651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9397.724280] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9397.935669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9403.872828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9403.876865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9403.893831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9403.904061] Lustre: Skipped 4 previous similar messages [ 9403.915735] Lustre: Skipped 3 previous similar messages [ 9414.112661] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9414.324252] Lustre: server umount lustre-OST0000 complete [ 9424.259799] Lustre: server umount lustre-MDT0000 complete [ 9424.352324] LustreError: 220238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9424.373406] LustreError: 220238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9427.784294] LustreError: 219670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464633 with bad export cookie 6391041782465989060 [ 9427.790986] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9427.802066] LustreError: 219670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9428.420019] Lustre: server umount lustre-MDT0001 complete [ 9437.962952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.359369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9438.366984] Lustre: Skipped 5 previous similar messages [ 9444.026735] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9454.528125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.600586] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9476.685377] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9486.835759] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9497.047601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9507.092754] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9516.839249] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9528.293264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9528.297537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9528.300410] LustreError: Skipped 1 previous similar message [ 9528.302770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9528.311528] Lustre: Skipped 3 previous similar messages [ 9528.324482] Lustre: Skipped 8 previous similar messages [ 9533.700715] Lustre: server umount lustre-OST0000 complete [ 9543.869530] Lustre: server umount lustre-MDT0000 complete [ 9547.389459] LustreError: 223795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788464753 with bad export cookie 6391041782465989935 [ 9547.397767] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9547.624626] Lustre: server umount lustre-MDT0001 complete [ 9658.006318] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 9661.723987] Key type lgssc unregistered [ 9662.105899] LNet: 226725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9662.136032] LNetError: 226725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9663.217635] LNet: Removed LNI 192.168.204.121@tcp [ 9664.458156] Key type .llcrypt unregistered [ 9664.459811] Key type ._llcrypt unregistered [ 9694.179228] Key type ._llcrypt registered [ 9694.182539] Key type .llcrypt registered [ 9694.293982] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 9711.418633] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9712.626815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9712.877392] alg: No test for adler32 (adler32-zlib) [ 9714.350319] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9714.823588] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 9716.599437] Key type lgssc registered [ 9717.577899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9757.117524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9757.138766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.386882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9758.452640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9758.602608] Lustre: lustre-MDT0000: new disk, initializing [ 9758.692268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9758.714732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9763.769228] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9775.876850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9776.118810] Lustre: 230078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9776.215590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9776.219610] Lustre: Skipped 1 previous similar message [ 9776.335784] Lustre: lustre-MDT0001: new disk, initializing [ 9776.463374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9776.507227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9776.521254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9781.458291] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9792.769808] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9800.653512] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9810.556800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9811.210909] Lustre: lustre-OST0000: new disk, initializing [ 9811.228251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9811.255984] Lustre: 232047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9811.497230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9819.210996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9819.232436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9819.413497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9819.730543] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9831.083850] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9836.569859] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9836.783321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9842.296403] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9842.446940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9845.216220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9845.216397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9845.228220] LustreError: Skipped 1 previous similar message [ 9845.230470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9845.247219] Lustre: Skipped 1 previous similar message [ 9845.266628] Lustre: Skipped 1 previous similar message [ 9850.342615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9850.366037] Lustre: Skipped 1 previous similar message [ 9850.919195] Lustre: server umount lustre-OST0000 complete [ 9855.466545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9855.473361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9855.493231] Lustre: Skipped 1 previous similar message [ 9855.511272] Lustre: Skipped 1 previous similar message [ 9860.577633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9860.591410] Lustre: Skipped 1 previous similar message [ 9861.600598] Lustre: server umount lustre-MDT0000 complete [ 9866.057267] LustreError: 230069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465071 with bad export cookie 382212473682182229 [ 9866.071544] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9866.077821] LustreError: 230069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9866.675940] Lustre: server umount lustre-MDT0001 complete [ 9890.714583] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 9904.774627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9905.487599] LustreError: 234982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9905.515517] LustreError: 234982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9905.576716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9910.753695] LustreError: 234983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9912.397232] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9915.884173] LustreError: 234982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9921.003821] LustreError: 234983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.054964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9924.458788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9930.516495] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9933.585493] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9940.691389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9941.163153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9949.099332] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9959.272618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9959.590510] Lustre: lustre-OST0001: new disk, initializing [ 9959.594787] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9959.625456] Lustre: 237290:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9959.811118] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9966.584709] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 9968.378637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9968.404047] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9968.548896] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9980.780783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9991.857217] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10008.882612] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 15:53:32 (1788465212) [10011.616475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10011.623826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10011.651360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10014.728293] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10014.745373] Lustre: Skipped 2 previous similar messages [10017.993599] Lustre: server umount lustre-MDT0000 complete [10019.814474] LustreError: 234983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10019.841226] LustreError: 234983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10022.417521] LustreError: 237289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465228 with bad export cookie 382212473682183118 [10022.418423] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10022.450499] LustreError: 237289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10023.224427] Lustre: server umount lustre-MDT0001 complete [10038.896856] Lustre: server umount lustre-OST0000 complete [10040.876996] Lustre: 228299:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788465230/real 1788465230] req@ffff9160f5865500 x1875341392074496/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788465246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10040.926741] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10043.903901] Lustre: server umount lustre-OST0001 complete [10051.376317] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [10061.101796] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [10103.018558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10103.227368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10103.258504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10103.333787] Lustre: lustre-MDT0000: new disk, initializing [10103.433235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10103.476899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10108.899645] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10121.155970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10121.345636] Lustre: 241242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10121.386120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10121.393134] Lustre: Skipped 1 previous similar message [10121.512615] Lustre: lustre-MDT0001: new disk, initializing [10121.733900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10121.744413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10127.129199] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10139.700267] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10146.670053] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10155.178429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10155.684226] Lustre: lustre-OST0000: new disk, initializing [10155.692812] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10155.702563] Lustre: 243209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10157.040895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10157.071967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10157.270472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10163.426825] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10174.738435] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10181.165734] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10181.392866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10187.069618] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10187.386738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10192.865988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10192.883948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10192.912746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10192.919751] Lustre: Skipped 4 previous similar messages [10195.751561] Lustre: server umount lustre-OST0000 complete [10203.622890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10203.633297] LustreError: Skipped 1 previous similar message [10205.856806] Lustre: server umount lustre-MDT0000 complete [10208.741874] LustreError: 241253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10208.777314] LustreError: 241253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10210.038490] LustreError: 241234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465415 with bad export cookie 382212473682185253 [10210.055530] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10210.061211] LustreError: 241234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10210.377714] Lustre: server umount lustre-MDT0001 complete [10219.449484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10220.142811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10220.154947] Lustre: Skipped 2 previous similar messages [10225.650805] LustreError: 245428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10225.679689] LustreError: 245428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10225.805810] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10236.343652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10242.643917] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10252.780887] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10262.336634] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10273.329042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10283.085355] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10291.328876] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10440.673306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10440.681368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10440.698532] Lustre: Skipped 3 previous similar messages [10440.710320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10440.714528] Lustre: Skipped 4 previous similar messages [10445.062933] Lustre: server umount lustre-OST0000 complete [10455.027539] Lustre: server umount lustre-MDT0000 complete [10458.684895] LustreError: 245408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465664 with bad export cookie 382212473682186135 [10458.701052] LustreError: 245408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10458.701231] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10458.991636] Lustre: server umount lustre-MDT0001 complete [10558.575959] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [10561.543404] Key type lgssc unregistered [10561.892719] LNet: 249048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10561.909967] LNetError: 249048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10561.928716] LNet: Removed LNI 192.168.204.121@tcp [10563.061183] Key type .llcrypt unregistered [10563.063661] Key type ._llcrypt unregistered [10590.584137] Key type ._llcrypt registered [10590.587038] Key type .llcrypt registered [10590.722528] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [10608.251435] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [10609.553732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10609.595379] alg: No test for adler32 (adler32-zlib) [10610.885092] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10611.260655] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [10612.930883] Key type lgssc registered [10614.622130] Lustre: Echo OBD driver; http://www.lustre.org/ [10668.961452] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [10680.122460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10680.145364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.431370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10681.476301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10681.564814] Lustre: lustre-MDT0000: new disk, initializing [10681.647311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10681.666983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10686.690114] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10699.008696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10699.190510] Lustre: 253117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10699.262120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10699.266932] Lustre: Skipped 1 previous similar message [10699.323922] Lustre: lustre-MDT0001: new disk, initializing [10699.420030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10699.466456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10699.481391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10703.721800] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10708.685094] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10715.922713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10716.293321] Lustre: lustre-OST0000: new disk, initializing [10716.298331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10716.320369] Lustre: 254748:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10716.445751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10716.763800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10716.779936] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10717.014506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10723.055976] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10733.684644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10733.874168] Lustre: lustre-OST0001: new disk, initializing [10733.878424] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10733.886316] Lustre: 255622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10733.999763] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10740.544452] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10742.350114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10742.364472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10742.484940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10753.260429] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10764.043839] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10781.664663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10781.670509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10781.686411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10783.714674] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10783.731566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10783.740530] Lustre: Skipped 2 previous similar messages [10783.752972] Lustre: Skipped 3 previous similar messages [10787.847843] Lustre: server umount lustre-MDT0000 complete [10792.305222] LustreError: 253111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788465997 with bad export cookie 16893958936698003681 [10792.308363] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10792.334363] LustreError: 253111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10793.295478] Lustre: server umount lustre-MDT0001 complete [10808.859649] Lustre: server umount lustre-OST0000 complete [10809.633324] Lustre: 250624:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788465999/real 1788465999] req@ffff915fc27cb480 x1875342331559040/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788466015 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10809.694684] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10813.407586] Lustre: 250622:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788466003/real 1788466003] req@ffff915fc27c9c00 x1875342331559296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788466019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10814.718164] Lustre: server umount lustre-OST0001 complete [10833.892147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10845.599871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10853.424562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10863.666953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10887.996818] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [10900.909289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10900.934940] Lustre: lustre-MDT0000: reset Object Index mappings [10901.534775] LustreError: 260638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10901.551686] LustreError: 260638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10901.753081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10907.049871] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10907.136754] LustreError: 260639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10912.229962] LustreError: 260639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10917.134345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10917.162739] Lustre: lustre-MDT0001: reset Object Index mappings [10917.347365] LustreError: 260638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10917.815072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10924.077948] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10927.461538] Lustre: 261780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10936.146996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10936.637034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10942.774972] LustreError: 262135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10943.689745] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10946.877124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10952.168766] LustreError: 262133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10952.207439] LustreError: 262133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10954.304038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10959.861277] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10962.064806] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [10970.094400] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10974.425364] Lustre: 263649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10984.421789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10984.437278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10984.466543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10990.567832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10990.577666] Lustre: Skipped 7 previous similar messages [10995.167938] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10995.460643] Lustre: server umount lustre-MDT0000 complete [10995.725432] LustreError: 261573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.744448] LustreError: 260618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466205 with bad export cookie 16893958936698010471 [10999.771893] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11000.370730] Lustre: server umount lustre-MDT0001 complete [11015.854268] Lustre: server umount lustre-OST0000 complete [11016.415115] Lustre: 250622:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788466206/real 1788466206] req@ffff9160ef39d180 x1875342331615360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788466222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11016.468124] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11016.484664] Lustre: Skipped 3 previous similar messages [11020.768539] Lustre: 250624:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788466210/real 1788466210] req@ffff915fcd465f80 x1875342331615616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788466226 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11021.229986] Lustre: server umount lustre-OST0001 complete [11045.767803] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:10:49 (1788466249) [11068.050174] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [11079.712749] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11120.300540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11120.555711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11120.583089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11120.674943] Lustre: lustre-MDT0000: new disk, initializing [11120.789679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11120.799842] Lustre: Skipped 1 previous similar message [11120.820758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11125.617240] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11138.843929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11139.174778] Lustre: 267910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11139.243117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11139.246766] Lustre: Skipped 1 previous similar message [11139.378275] Lustre: lustre-MDT0001: new disk, initializing [11139.518694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11139.537448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11144.548661] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11154.496439] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11161.652824] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11169.796428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11170.027219] Lustre: lustre-OST0000: new disk, initializing [11170.032115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11170.041538] Lustre: 269882:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11171.719170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11171.740120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11171.861942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11176.593432] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11187.359820] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11192.738961] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11192.914452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11197.792172] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11198.038797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11202.533563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11202.536616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11202.549661] LustreError: Skipped 1 previous similar message [11202.554578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11202.559503] Lustre: Skipped 1 previous similar message [11202.575759] Lustre: Skipped 1 previous similar message [11212.774217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11212.786727] Lustre: Skipped 3 previous similar messages [11214.303100] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11214.490327] Lustre: server umount lustre-OST0000 complete [11221.473097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11221.490526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11224.949586] Lustre: server umount lustre-MDT0000 complete [11228.131334] LustreError: 267923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11228.165083] LustreError: 267923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11229.192238] LustreError: 267903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466434 with bad export cookie 16893958936698012564 [11229.205396] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11229.209232] LustreError: 267903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11229.567828] Lustre: server umount lustre-MDT0001 complete [11253.848919] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11264.576513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11265.288192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11265.302589] Lustre: Skipped 2 previous similar messages [11269.373358] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11278.240808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11283.082678] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11286.039729] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11292.583886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11299.164577] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11308.965825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11309.151185] Lustre: lustre-OST0001: new disk, initializing [11309.154777] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11309.163536] Lustre: 275123:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11310.688046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11310.705707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11310.813122] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11317.035679] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11327.670814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11331.626298] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11337.453829] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11337.536608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11350.496470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11350.505071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11350.522949] Lustre: Skipped 1 previous similar message [11350.530177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11350.540177] Lustre: Skipped 3 previous similar messages [11354.654848] Lustre: server umount lustre-MDT0000 complete [11357.170267] LustreError: 274084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11357.190088] LustreError: 274084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11358.524881] LustreError: 272797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466564 with bad export cookie 16893958936698013446 [11358.528071] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11358.537277] LustreError: 272797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11359.046708] Lustre: server umount lustre-MDT0001 complete [11373.457857] Lustre: server umount lustre-OST0000 complete [11387.215905] Lustre: server umount lustre-OST0001 complete [11405.307345] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11415.220110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11415.623214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11415.630536] Lustre: Skipped 3 previous similar messages [11420.170548] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11429.556149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11435.837108] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11439.557663] Lustre: 279613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11447.146530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11454.760958] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11461.112253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11463.497348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11469.309514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11470.105282] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11478.786536] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11482.630693] Lustre: 281492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11488.421852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11488.490316] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11505.837362] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:18:29 (1788466709) [11522.528690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11522.537564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11522.548671] Lustre: Skipped 3 previous similar messages [11522.563364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11522.576185] Lustre: Skipped 4 previous similar messages [11536.351687] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11536.592421] Lustre: server umount lustre-MDT0000 complete [11540.971211] LustreError: 280338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11541.003370] LustreError: 280338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11541.885079] LustreError: 278453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466747 with bad export cookie 16893958936698020299 [11541.888601] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11541.894513] LustreError: 278453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11556.319247] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11556.788732] Lustre: server umount lustre-MDT0001 complete [11569.710480] Lustre: server umount lustre-OST0000 complete [11574.119828] Lustre: server umount lustre-OST0001 complete [11581.403712] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [11592.958470] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11646.502864] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [11658.849447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11659.111479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11659.136558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11659.218275] Lustre: lustre-MDT0000: new disk, initializing [11659.296465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11665.315034] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11678.114721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11678.307860] Lustre: 286025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11678.671801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11678.684508] Lustre: Skipped 4 previous similar messages [11678.750223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11683.379771] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11695.199136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11695.520548] Lustre: 287657:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11697.841286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11702.156730] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11713.216576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11713.416131] Lustre: 288528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11715.335636] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11715.347727] Lustre: Skipped 1 previous similar message [11715.462511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11720.787897] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11730.310545] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11734.642272] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11734.645541] Lustre: Skipped 1 previous similar message [11750.030893] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:22:33 (1788466953) [11754.397906] Lustre: server umount lustre-MDT0000 complete [11755.506217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11755.514771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11755.533043] Lustre: Skipped 5 previous similar messages [11758.048771] LustreError: 286017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788466963 with bad export cookie 16893958936698027775 [11758.054124] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11758.062710] LustreError: 286017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11758.356543] Lustre: server umount lustre-MDT0001 complete [11771.509922] Lustre: server umount lustre-OST0000 complete [11785.939864] Lustre: server umount lustre-OST0001 complete [11795.153169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.007840] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11801.063790] LustreError: 290777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.106874] LustreError: 290777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [11809.672186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11814.605717] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11822.198180] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11828.892893] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11835.514474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11842.164757] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11850.140713] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11868.888490] loop1: detected capacity change from 0 to 204800000 [11869.137786] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11869.155221] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11869.168727] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11874.801303] loop1: detected capacity change from 0 to 204800000 [11880.996255] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11881.061770] LustreError: 290766:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs1953: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11881.088273] LustreError: 290766:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs1953 [11881.095747] LustreError: 290766:0:(mgs_handler.c:517:mgs_target_reg()) fs1953-MDT0001: Error checking target index: rc = -13 [11881.107542] LustreError: fs1953-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11881.121625] LustreError: 294494:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11881.135023] LustreError: 294494:0:(tgt_mount.c:2132:server_put_super()) no obd fs1953-MDT0001 [11881.141857] LustreError: 294494:0:(tgt_mount.c:116:server_deregister_mount()) fs1953-MDT0001 not registered [11881.226604] Lustre: server umount fs1953-MDT0001 complete [11881.229355] LustreError: 294494:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11890.847859] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11891.106994] Lustre: Setting parameter fs1953-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs1953-MDT0001 [11891.163377] Lustre: srv-fs1953-MDT0001: No data found on store. Initialize space. [11891.171412] Lustre: Skipped 4 previous similar messages [11891.285272] Lustre: fs1953-MDT0001: new disk, initializing [11891.289584] Lustre: Skipped 3 previous similar messages [11891.412091] Lustre: srv-fs1953-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11891.420199] Lustre: 294962:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1953-MDT0001-mdtlov: can't create new llog: rc = -115 [11891.431648] LustreError: 294962:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1953-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11896.043994] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11902.910301] Lustre: server umount fs1953-MDT0001 complete [11920.192053] loop1: detected capacity change from 0 to 204800000 [11921.414218] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11927.422995] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11928.184028] Lustre: 296529:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1953-MDT0001-mdtlov: can't create new llog: rc = -115 [11928.203477] LustreError: 296529:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1953-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11934.209466] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11939.724886] Lustre: server umount fs1953-MDT0001 complete [11952.104376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11952.122635] Lustre: Skipped 19 previous similar messages [11964.384886] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11964.630107] Lustre: server umount lustre-MDT0000 complete [11973.536460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11973.676395] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11978.642707] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [11979.252464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11986.624798] loop1: detected capacity change from 0 to 204800000 [11991.995134] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11992.385971] Lustre: 298979:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs1953-MDT0001-mdtlov: can't create new llog: rc = -115 [11992.397151] LustreError: 298979:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs1953-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11996.042194] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12000.185693] Lustre: server umount fs1953-MDT0001 complete [12023.780264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12023.788327] LustreError: Skipped 1 previous similar message [12023.794194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12023.814985] Lustre: Skipped 6 previous similar messages [12026.976275] Lustre: server umount lustre-OST0000 complete [12036.988524] Lustre: server umount lustre-MDT0000 complete [12040.386984] LustreError: 290757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788467246 with bad export cookie 16893958936698030463 [12040.393685] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12040.400258] LustreError: 290757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12040.720110] Lustre: server umount lustre-MDT0001 complete [12051.773270] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [12054.780966] Key type lgssc unregistered [12055.048047] LNet: 301322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12055.053768] LNetError: 301322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12055.075380] LNet: Removed LNI 192.168.204.121@tcp [12056.026220] Key type .llcrypt unregistered [12056.030175] Key type ._llcrypt unregistered [12077.271836] Key type ._llcrypt registered [12077.274632] Key type .llcrypt registered [12077.441094] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [12078.651113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12078.710311] alg: No test for adler32 (adler32-zlib) [12079.852606] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12080.082801] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [12081.743322] Key type lgssc registered [12083.246226] Lustre: Echo OBD driver; http://www.lustre.org/ [12090.465566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12090.493302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12091.846311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12095.305417] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12102.112448] LustreError: 302592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12102.132484] LustreError: 302592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12103.114137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12103.491588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12107.841649] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12114.399393] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12119.565208] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12125.657599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12126.061608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12129.138818] LustreError: 304423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12131.332626] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12134.389824] LustreError: 304421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12136.423322] LustreError: 304423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12136.446627] LustreError: 304423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12137.409796] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12141.539105] LustreError: 305047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12141.564384] LustreError: 305047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12150.314867] LustreError: 305046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12150.327658] LustreError: 305046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12153.774410] loop1: detected capacity change from 0 to 204800000 [12154.070156] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12154.080157] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12154.092801] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12158.502279] loop1: detected capacity change from 0 to 204800000 [12163.014638] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12163.182545] Lustre: fs21296-OST0000: new disk, initializing [12163.188734] Lustre: srv-fs21296-OST0000: No data found on store. Initialize space. [12163.195734] Lustre: 306327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12163.281586] Lustre: fs21296-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12167.142405] LustreError: 304421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12167.154071] LustreError: 304421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [12168.332891] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12175.852309] Lustre: server umount fs21296-OST0000 complete [12185.456494] loop1: detected capacity change from 0 to 204800000 [12189.728763] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12189.898266] Lustre: fs21296-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12194.647170] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12201.495904] LustreError: 304421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12201.518356] LustreError: 304421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [12202.591214] loop2: detected capacity change from 0 to 204800000 [12202.779953] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12202.796856] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12202.804784] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12207.251622] loop2: detected capacity change from 0 to 204800000 [12211.664198] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12211.726752] LustreError: fs21296-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12211.736544] LustreError: fs21296-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12211.745556] LustreError: 309366:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [12211.750432] LustreError: 309366:0:(tgt_mount.c:2132:server_put_super()) no obd fs21296-OST0001 [12211.754178] LustreError: 309366:0:(tgt_mount.c:116:server_deregister_mount()) fs21296-OST0001 not registered [12211.782506] Lustre: server umount fs21296-OST0001 complete [12211.784690] LustreError: 309366:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12213.225338] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12218.342195] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12219.700731] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12219.812782] Lustre: fs21296-OST0001: new disk, initializing [12219.816874] Lustre: srv-fs21296-OST0001: No data found on store. Initialize space. [12219.822203] Lustre: 309916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12219.887124] Lustre: fs21296-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12225.286336] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12232.872971] Lustre: server umount fs21296-OST0000 complete [12242.314623] Lustre: server umount fs21296-OST0001 complete [12254.363486] loop1: detected capacity change from 0 to 204800000 [12258.937667] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12259.087991] Lustre: fs21296-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12264.057341] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12268.057232] LustreError: 305047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.077088] LustreError: 305047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [12269.662941] loop2: detected capacity change from 0 to 204800000 [12275.338569] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12281.488411] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12290.696900] Lustre: server umount fs21296-OST0000 complete [12299.949043] Lustre: server umount fs21296-OST0001 complete [12312.544059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12312.549175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12312.558479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12315.627780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12315.630878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12315.635935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12315.660035] Lustre: Skipped 1 previous similar message [12320.740286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12320.757232] Lustre: Skipped 1 previous similar message [12325.859127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12325.870394] Lustre: Skipped 1 previous similar message [12326.367476] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12326.506280] Lustre: server umount lustre-OST0000 complete [12330.982289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12330.996612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12331.010366] Lustre: Skipped 1 previous similar message [12331.035740] Lustre: Skipped 1 previous similar message [12336.413780] Lustre: server umount lustre-MDT0000 complete [12341.133638] LustreError: 302573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788467546 with bad export cookie 2868958878395653794 [12341.134161] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12341.147108] LustreError: 302573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12341.491532] Lustre: server umount lustre-MDT0001 complete [12353.522582] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [12356.075954] Key type lgssc unregistered [12356.349330] LNet: 315836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12356.361577] LNetError: 315836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12356.384982] LNet: Removed LNI 192.168.204.121@tcp [12357.398164] Key type .llcrypt unregistered [12357.399905] Key type ._llcrypt unregistered [12386.848300] Key type ._llcrypt registered [12386.850575] Key type .llcrypt registered [12386.944291] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [12400.969446] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [12402.148155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12402.171340] alg: No test for adler32 (adler32-zlib) [12403.214508] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12403.407825] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [12405.039167] Key type lgssc registered [12406.086235] Lustre: Echo OBD driver; http://www.lustre.org/ [12437.118067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12437.134535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12438.301285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12438.336560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12438.405496] Lustre: lustre-MDT0000: new disk, initializing [12438.448565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12438.470790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12441.203836] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12448.950238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12449.040055] Lustre: 319695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12449.089757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12449.094210] Lustre: Skipped 1 previous similar message [12449.153095] Lustre: lustre-MDT0001: new disk, initializing [12449.217746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12449.246743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12449.256181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12452.168416] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12459.646454] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12464.242259] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12468.821666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12469.030454] Lustre: lustre-OST0000: new disk, initializing [12469.033946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12469.039730] Lustre: 321661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12469.109398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12474.206777] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12478.488258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12478.500230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12478.565602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12481.794423] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12483.473290] Lustre: Mounted lustre-client - version 2.17.58_2_gee80f9b [12484.375759] Lustre: Unmounted lustre-client [12493.794059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12493.797805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12493.805919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12493.812019] LustreError: Skipped 1 previous similar message [12493.817580] Lustre: Skipped 1 previous similar message [12498.369348] Lustre: server umount lustre-OST0000 complete [12504.034054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12504.045153] Lustre: Skipped 1 previous similar message [12504.050081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12504.055551] Lustre: Skipped 1 previous similar message [12509.155348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12509.163169] Lustre: Skipped 1 previous similar message [12514.276131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12514.283892] Lustre: Skipped 1 previous similar message [12515.295141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12515.464085] Lustre: server umount lustre-MDT0000 complete [12518.345721] LustreError: 319687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788467723 with bad export cookie 3657221080916369506 [12518.347444] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12518.352738] LustreError: 319687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12518.598068] Lustre: server umount lustre-MDT0001 complete [12528.117714] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [12530.565631] Key type lgssc unregistered [12533.473432] LNet: 1 local NIs in recovery (showing 1): 192.168.204.121@tcp [12533.920591] LNet: 323825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12533.929808] LNetError: 323825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12612.787618] LNet: Removed LNI 192.168.204.121@tcp [12613.882749] Key type .llcrypt unregistered [12613.885503] Key type ._llcrypt unregistered [12635.914603] Key type ._llcrypt registered [12635.917268] Key type .llcrypt registered [12636.006721] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [12653.862644] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [12654.943538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12655.038078] alg: No test for adler32 (adler32-zlib) [12656.042746] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12656.370770] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [12658.135426] Key type lgssc registered [12659.445317] Lustre: Echo OBD driver; http://www.lustre.org/ [12695.127713] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:38:18 (1788467898) [12702.100806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12702.154354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12703.424947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12703.459528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12703.559117] Lustre: lustre-MDT0000: new disk, initializing [12703.657367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12703.675488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12708.157601] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12719.176790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12719.326359] Lustre: 327042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12719.384535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12719.388543] Lustre: Skipped 1 previous similar message [12719.441974] Lustre: lustre-MDT0001: new disk, initializing [12719.511356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12719.545416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12719.558897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12724.071390] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12734.280676] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12741.050751] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12748.406224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12748.656280] Lustre: lustre-OST0000: new disk, initializing [12748.660874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12748.668418] Lustre: 329014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12748.762053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12754.963391] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12756.522868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12756.532461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12756.625244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12764.147781] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12782.052080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12782.058246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12782.067755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12782.575140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12782.583131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12782.594046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12784.581853] Lustre: server umount lustre-OST0000 complete [12791.268971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12791.275144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12791.296844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12795.466880] Lustre: server umount lustre-MDT0000 complete [12800.031556] LustreError: 329013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468005 with bad export cookie 1238487776796120479 [12800.041086] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12800.046976] LustreError: 329013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12800.409315] Lustre: server umount lustre-MDT0001 complete [12813.396608] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [12816.439801] Key type lgssc unregistered [12816.914779] LNet: 330923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12816.928023] LNetError: 330923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12816.953741] LNet: Removed LNI 192.168.204.121@tcp [12817.807649] Key type .llcrypt unregistered [12817.810154] Key type ._llcrypt unregistered [12837.200459] Key type ._llcrypt registered [12837.207698] Key type .llcrypt registered [12837.297547] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [12838.139942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12838.226630] alg: No test for adler32 (adler32-zlib) [12839.418782] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12839.680355] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [12841.335290] Key type lgssc registered [12842.520703] Lustre: Echo OBD driver; http://www.lustre.org/ [12850.593617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12850.608694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12851.994513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12857.395703] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12861.411860] LustreError: 332194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12861.428355] LustreError: 332194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12866.530498] LustreError: 332195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12866.572531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12867.172094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12872.135317] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12878.697951] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12883.563354] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12889.805801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12890.144223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12897.065488] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12904.016750] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12913.452955] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12914.813412] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12921.905263] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.21@tcp (stopping) [12922.858092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12922.874852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12922.877643] Lustre: Skipped 1 previous similar message [12922.901551] Lustre: Skipped 2 previous similar messages [12925.488468] Lustre: server umount lustre-MDT0000 complete [12927.005272] LustreError: 332928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12930.409104] LustreError: 332174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468136 with bad export cookie 14647063894267566584 [12930.421643] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12930.447199] LustreError: 332174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12932.121180] LustreError: 332191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12932.169034] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (stopping) [12932.205517] LustreError: 332191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12933.099791] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12933.124701] Lustre: Skipped 1 previous similar message [12935.138216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12935.150856] Lustre: Skipped 1 previous similar message [12936.908864] Lustre: server umount lustre-MDT0001 complete [12941.295645] Lustre: server umount lustre-OST0000 complete [12952.177640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12958.099220] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12960.965455] Lustre: 336377:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.121@tcp [12963.524551] Lustre: server umount MGS complete [12973.025498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12973.487379] LustreError: 336948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12973.511113] LustreError: 336948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12973.588907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12979.429752] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [12983.778942] LustreError: 336949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12983.804876] LustreError: 336949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12989.664817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12990.333336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12995.683157] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13006.741201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13007.470052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13014.963734] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13024.146675] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13025.971055] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13032.522731] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13034.545908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13040.611616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13040.622322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13040.647566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13041.131816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13042.074241] Lustre: server umount lustre-OST0000 complete [13046.755408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13046.766262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13046.784757] Lustre: Skipped 1 previous similar message [13049.824061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13049.828989] Lustre: Skipped 2 previous similar messages [13052.472312] Lustre: server umount lustre-MDT0000 complete [13054.949994] LustreError: 336949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13054.970198] LustreError: 336949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13056.677553] LustreError: 336928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468262 with bad export cookie 14647063894267567424 [13056.690647] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13056.698772] LustreError: 336928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13057.206862] Lustre: server umount lustre-MDT0001 complete [13071.187640] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [13074.441234] Key type lgssc unregistered [13074.776906] LNet: 340388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13074.786883] LNetError: 340388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13074.807836] LNet: Removed LNI 192.168.204.121@tcp [13075.723866] Key type .llcrypt unregistered [13075.726610] Key type ._llcrypt unregistered [13095.289313] Key type ._llcrypt registered [13095.293469] Key type .llcrypt registered [13095.435605] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [13115.167766] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13115.747872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13115.782289] alg: No test for adler32 (adler32-zlib) [13117.175341] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13117.501620] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [13119.231298] Key type lgssc registered [13121.548210] Lustre: Echo OBD driver; http://www.lustre.org/ [13169.062072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13169.087956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13170.542973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13170.610087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13170.774726] Lustre: lustre-MDT0000: new disk, initializing [13170.913545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13170.949408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13177.565421] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13190.786887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13190.960607] Lustre: 343549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13191.036993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13191.041819] Lustre: Skipped 1 previous similar message [13191.111410] Lustre: lustre-MDT0001: new disk, initializing [13191.218621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13191.265384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13191.279572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13196.759493] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13208.179704] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13214.297936] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13222.992733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13223.254338] Lustre: lustre-OST0000: new disk, initializing [13223.260211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13223.268416] Lustre: 345517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13223.382875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13230.381435] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13230.675854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13230.698980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13230.804797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13241.044151] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13246.344094] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13246.572616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13251.775043] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13251.931712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13256.694510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13256.697175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13256.702521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13256.730367] Lustre: Skipped 1 previous similar message [13260.088421] Lustre: server umount lustre-OST0000 complete [13266.912633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13266.940256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13266.960046] Lustre: Skipped 2 previous similar messages [13270.686666] Lustre: server umount lustre-MDT0000 complete [13274.680380] LustreError: 343540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468480 with bad export cookie 1394666720912628373 [13274.686192] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13274.691250] LustreError: 343540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13275.135189] Lustre: server umount lustre-MDT0001 complete [13287.467313] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:48:10 (1788468490) [13310.820251] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [13329.111166] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13369.992301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13370.390653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13370.466245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13370.687440] Lustre: lustre-MDT0000: new disk, initializing [13370.865140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13370.889671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13377.107412] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13390.584538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13390.727740] Lustre: 350302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13390.813661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13390.819639] Lustre: Skipped 1 previous similar message [13390.908495] Lustre: lustre-MDT0001: new disk, initializing [13391.045281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13391.097139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13391.125784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13397.981347] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13409.736528] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13416.785147] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13425.750082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13426.146204] Lustre: lustre-OST0000: new disk, initializing [13426.150602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13426.156867] Lustre: 352269:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13426.269193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13433.809965] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13436.468405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13436.483062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13436.587178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13443.571879] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13488.106233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13488.108409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13488.129070] Lustre: Skipped 1 previous similar message [13488.134671] LustreError: Skipped 1 previous similar message [13488.143153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13488.152492] Lustre: Skipped 1 previous similar message [13493.236381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13493.248888] Lustre: Skipped 1 previous similar message [13498.336444] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13498.348284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13498.363021] Lustre: Skipped 1 previous similar message [13498.637864] Lustre: server umount lustre-OST0000 complete [13503.971801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13503.975869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13508.579155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13508.579678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13508.593762] Lustre: Skipped 1 previous similar message [13509.833304] Lustre: server umount lustre-MDT0000 complete [13513.705480] LustreError: 350312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13513.723460] LustreError: 350312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13513.726282] LustreError: 350292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468719 with bad export cookie 1394666720912629255 [13513.729754] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13513.778702] LustreError: 350292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13514.227700] Lustre: server umount lustre-MDT0001 complete [13527.110636] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [13530.020242] Key type lgssc unregistered [13530.325349] LNet: 354444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13530.341924] LNetError: 354444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13530.371389] LNet: Removed LNI 192.168.204.121@tcp [13531.252786] Key type .llcrypt unregistered [13531.258612] Key type ._llcrypt unregistered [13553.149675] Key type ._llcrypt registered [13553.156153] Key type .llcrypt registered [13553.255334] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13554.182926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13554.292685] alg: No test for adler32 (adler32-zlib) [13555.326917] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13555.564217] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [13557.255208] Key type lgssc registered [13558.592153] Lustre: Echo OBD driver; http://www.lustre.org/ [13566.922691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13566.975633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13568.421408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13574.498891] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13578.726936] LustreError: 355717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13578.752116] LustreError: 355717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13583.841580] LustreError: 355716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13583.992378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13584.488901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13588.831975] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13597.533440] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13603.719551] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13612.369786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13612.844195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13621.145946] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13631.119732] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13645.805464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13645.824880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13645.827923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13650.920326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13650.943813] Lustre: Skipped 1 previous similar message [13651.947352] Lustre: server umount lustre-OST0000 complete [13661.154070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13661.174562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13661.201610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13661.205533] Lustre: Skipped 2 previous similar messages [13661.218037] Lustre: Skipped 2 previous similar messages [13664.336687] Lustre: server umount lustre-MDT0000 complete [13666.276266] LustreError: 356445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13666.301558] LustreError: 356445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13669.558434] LustreError: 355697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468875 with bad export cookie 5894150444951311201 [13669.560848] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13669.575961] LustreError: 355697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13670.015954] Lustre: server umount lustre-MDT0001 complete [13684.423800] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [13688.085949] Key type lgssc unregistered [13688.537247] LNet: 361206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13688.554964] LNetError: 361206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13688.577800] LNet: Removed LNI 192.168.204.121@tcp [13689.736544] Key type .llcrypt unregistered [13689.741854] Key type ._llcrypt unregistered [13710.440347] Key type ._llcrypt registered [13710.443673] Key type .llcrypt registered [13710.556724] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13711.905424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13711.942153] alg: No test for adler32 (adler32-zlib) [13713.041854] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13713.295459] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [13715.000118] Key type lgssc registered [13716.422713] Lustre: Echo OBD driver; http://www.lustre.org/ [13725.035527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13725.088899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13726.589160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13731.025155] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13736.937708] LustreError: 365152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13736.958764] LustreError: 365152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13740.527417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13741.097733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13746.460517] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13756.955744] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13766.336092] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13777.735808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13778.198852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13785.737576] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13794.916599] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13811.180871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13811.197835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13811.211926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13812.197949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13812.203107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13812.216933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13812.243587] Lustre: Skipped 1 previous similar message [13815.425248] Lustre: server umount lustre-OST0000 complete [13822.461771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13822.469970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13822.478519] Lustre: Skipped 1 previous similar message [13822.495725] Lustre: Skipped 1 previous similar message [13826.758348] Lustre: server umount lustre-MDT0000 complete [13827.557939] LustreError: 365153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13827.581292] LustreError: 365153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13831.450975] LustreError: 366504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469037 with bad export cookie 2862577469510792923 [13831.453073] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13831.461404] LustreError: 366504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13832.213175] Lustre: server umount lustre-MDT0001 complete [13846.197823] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [13850.586400] Key type lgssc unregistered [13851.276920] LNet: 372639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13851.286528] LNetError: 372639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13851.328197] LNet: Removed LNI 192.168.204.121@tcp [13852.867580] Key type .llcrypt unregistered [13852.870283] Key type ._llcrypt unregistered [13875.204531] Key type ._llcrypt registered [13875.208158] Key type .llcrypt registered [13875.411084] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [13877.406700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13877.440363] alg: No test for adler32 (adler32-zlib) [13878.560207] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13878.962565] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [13880.744295] Key type lgssc registered [13882.869104] Lustre: Echo OBD driver; http://www.lustre.org/ [13895.042296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13895.140609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13897.250086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13903.933459] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13906.452628] LustreError: 376391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13906.487767] LustreError: 376391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13911.535773] LustreError: 376392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13916.640118] LustreError: 376391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13918.925758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13919.855529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13927.098938] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13936.694357] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13945.699285] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13958.597210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13959.936857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13973.601445] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [13986.455422] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14001.121339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14001.134358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14001.174745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14004.217492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14004.225465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14004.242282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14006.185849] Lustre: server umount lustre-OST0000 complete [14014.439957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14014.446332] Lustre: Skipped 1 previous similar message [14014.446961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14014.451279] Lustre: Skipped 1 previous similar message [14019.555553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14019.566478] Lustre: Skipped 2 previous similar messages [14024.672569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14024.676864] Lustre: Skipped 1 previous similar message [14027.231479] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14027.662984] Lustre: server umount lustre-MDT0000 complete [14029.797638] LustreError: 376383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14029.846865] LustreError: 376383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14032.658525] LustreError: 376345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469238 with bad export cookie 14900592565261666417 [14032.680352] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14033.070682] Lustre: server umount lustre-MDT0001 complete [14044.616978] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14048.467221] Key type lgssc unregistered [14048.834727] LNet: 382185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14048.843399] LNetError: 382185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14048.871029] LNet: Removed LNI 192.168.204.121@tcp [14049.955881] Key type .llcrypt unregistered [14049.966456] Key type ._llcrypt unregistered [14179.894899] Key type ._llcrypt registered [14179.900358] Key type .llcrypt registered [14180.024636] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14180.321585] Key type .llcrypt unregistered [14180.324674] Key type ._llcrypt unregistered [14270.394144] Key type ._llcrypt registered [14270.402209] Key type .llcrypt registered [14270.522074] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14270.789503] Key type .llcrypt unregistered [14270.794178] Key type ._llcrypt unregistered [14303.748870] Key type ._llcrypt registered [14303.756847] Key type .llcrypt registered [14303.912342] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14304.327530] Key type .llcrypt unregistered [14304.330917] Key type ._llcrypt unregistered [14346.058210] Key type ._llcrypt registered [14346.066231] Key type .llcrypt registered [14346.349718] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14347.014316] Key type .llcrypt unregistered [14347.021269] Key type ._llcrypt unregistered [14455.019519] Key type ._llcrypt registered [14455.026899] Key type .llcrypt registered [14455.162561] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14455.533772] Key type .llcrypt unregistered [14455.536473] Key type ._llcrypt unregistered [14489.528035] Key type ._llcrypt registered [14489.541464] Key type .llcrypt registered [14489.735870] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14490.111295] Key type .llcrypt unregistered [14490.113501] Key type ._llcrypt unregistered [14525.977085] Key type ._llcrypt registered [14525.979668] Key type .llcrypt registered [14526.162750] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14526.542641] Key type .llcrypt unregistered [14526.545875] Key type ._llcrypt unregistered [14561.475468] Key type ._llcrypt registered [14561.478781] Key type .llcrypt registered [14561.573988] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14562.027051] Key type .llcrypt unregistered [14562.030700] Key type ._llcrypt unregistered [14603.567935] Key type ._llcrypt registered [14603.572634] Key type .llcrypt registered [14603.700449] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14604.089744] Key type .llcrypt unregistered [14604.094383] Key type ._llcrypt unregistered [14639.113815] Key type ._llcrypt registered [14639.118442] Key type .llcrypt registered [14639.253542] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14639.507621] Key type .llcrypt unregistered [14639.514133] Key type ._llcrypt unregistered [14716.128665] Key type ._llcrypt registered [14716.130388] Key type .llcrypt registered [14716.275710] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14716.629335] Key type .llcrypt unregistered [14716.632605] Key type ._llcrypt unregistered [14753.746987] Key type ._llcrypt registered [14753.750588] Key type .llcrypt registered [14753.908329] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14754.452587] Key type .llcrypt unregistered [14754.455928] Key type ._llcrypt unregistered [14801.469701] Key type ._llcrypt registered [14801.473777] Key type .llcrypt registered [14801.827677] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [14802.527336] Key type .llcrypt unregistered [14802.531178] Key type ._llcrypt unregistered [14876.128200] Key type ._llcrypt registered [14876.133443] Key type .llcrypt registered [14876.322083] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [14897.993176] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [14899.130389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14899.201118] alg: No test for adler32 (adler32-zlib) [14900.416905] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14900.777213] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [14902.577992] Key type lgssc registered [14904.190616] Lustre: Echo OBD driver; http://www.lustre.org/ [14952.269571] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14952.312618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14953.536963] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14953.562567] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14953.658342] Lustre: tst250fs-MDT0000: new disk, initializing [14953.742290] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14953.763519] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14960.395950] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [14973.211252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14973.373303] Lustre: 402823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14973.480287] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14973.492740] Lustre: Skipped 1 previous similar message [14973.688642] Lustre: tst250fs-MDT0001: new disk, initializing [14973.801334] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14973.856494] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14973.874579] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14979.412830] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [14985.381495] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14995.004069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14995.534514] Lustre: tst250fs-OST0000: new disk, initializing [14995.547307] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14995.568234] Lustre: 404455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14995.850606] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15001.106909] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15001.128044] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15001.235308] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15004.300451] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [15020.107080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15020.296547] Lustre: tst250fs-OST0001: new disk, initializing [15020.303128] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [15020.310502] Lustre: 405327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15020.450477] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15030.465336] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15030.486894] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15030.642570] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15030.886874] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [15039.810548] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [15042.551754] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [15042.557387] Lustre: Skipped 1 previous similar message [15071.213732] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15071.230215] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15071.265126] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15071.734474] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15071.769265] Lustre: Skipped 2 previous similar messages [15071.788057] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15071.797930] Lustre: Skipped 2 previous similar messages [15073.883934] Lustre: server umount tst250fs-MDT0000 complete [15079.993150] LustreError: 402815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470285 with bad export cookie 13602796356092658385 [15080.002393] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15080.009312] LustreError: 402815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15080.712135] Lustre: server umount tst250fs-MDT0001 complete [15096.183442] Lustre: server umount tst250fs-OST0000 complete [15097.829855] Lustre: 401041:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470287/real 1788470287] req@ffff9160c57baa00 x1875346829948672/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15097.873643] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15100.383110] Lustre: 401043:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788470290/real 1788470290] req@ffff9160e67bed80 x1875346829948928/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788470306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15101.224699] Lustre: server umount tst250fs-OST0001 complete [15127.182182] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [15137.514957] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [15181.989897] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 17:19:45 (1788470385) [15184.762420] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [15209.422523] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [15228.839667] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [15269.973633] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14927 sec ========= 17:21:13 (1788470473) [15272.099163] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:21:15 (1788470475) === [15276.367433] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:21:19 (1788470479) === [15322.553707] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [15326.526574] Key type lgssc unregistered [15327.001802] LNet: 414496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15327.016160] LNetError: 414496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15327.037569] LNet: Removed LNI 192.168.204.121@tcp [15327.948143] Key type .llcrypt unregistered [15327.951347] Key type ._llcrypt unregistered