[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490683072 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007498] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010119] LSM: Security Framework initializing [ 0.011040] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013053] *** VALIDATE selinux *** [ 0.021323] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026534] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029092] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030125] *** VALIDATE tmpfs *** [ 0.032443] *** VALIDATE proc *** [ 0.033242] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035245] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038035] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.041642] debug: unmapping init [mem 0xffffffff8c059000-0xffffffff8c060fff] [ 0.043918] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044656] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046017] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051011] ... event mask: 000000070000000f [ 0.052313] rcu: Hierarchical SRCU implementation. [ 0.054548] smp: Bringing up secondary CPUs ... [ 0.055629] x86: Booting SMP configuration: [ 0.056027] .... node #0, CPUs: #1 #2 #3 [ 0.059296] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137023] node 0 deferred pages initialised in 73ms [ 0.140137] devtmpfs: initialized [ 0.141270] x86/mm: Memory block size: 128MB [ 0.143527] gcov: version magic: 0x41383552 [ 0.145407] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147301] pinctrl core: initialized pinctrl subsystem [ 0.148445] [ 0.148817] ************************************************************* [ 0.150014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153012] ** ** [ 0.155010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156012] ** ** [ 0.159016] ** This means that this kernel is built to expose internal ** [ 0.161014] ** IOMMU data structures, which may compromise security on ** [ 0.164015] ** your system. ** [ 0.166012] ** ** [ 0.169018] ** If you see this message and you are not debugging the ** [ 0.171014] ** kernel, report this immediately to your vendor! ** [ 0.174016] ** ** [ 0.176011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178015] ************************************************************* [ 0.182033] NET: Registered protocol family 16 [ 0.183444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.186075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.189073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.193114] cpuidle: using governor menu [ 0.194656] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.197239] PCI: Using configuration type 1 for base access [ 0.199134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216068] cryptd: max_cpu_qlen set to 1000 [ 0.218251] ACPI: Added _OSI(Module Device) [ 0.220021] ACPI: Added _OSI(Processor Device) [ 0.222011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235646] ACPI: Interpreter enabled [ 0.237063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.239022] ACPI: Using IOAPIC for interrupt routing [ 0.241129] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244399] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253965] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.257018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.261082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266303] acpiphp: Slot [2] registered [ 0.268127] acpiphp: Slot [5] registered [ 0.269074] acpiphp: Slot [6] registered [ 0.270126] acpiphp: Slot [7] registered [ 0.272122] acpiphp: Slot [8] registered [ 0.273099] acpiphp: Slot [9] registered [ 0.273940] acpiphp: Slot [10] registered [ 0.275106] acpiphp: Slot [3] registered [ 0.276091] acpiphp: Slot [4] registered [ 0.278091] acpiphp: Slot [11] registered [ 0.279113] acpiphp: Slot [12] registered [ 0.280076] acpiphp: Slot [13] registered [ 0.281075] acpiphp: Slot [14] registered [ 0.282080] acpiphp: Slot [15] registered [ 0.284117] acpiphp: Slot [16] registered [ 0.285106] acpiphp: Slot [17] registered [ 0.286088] acpiphp: Slot [18] registered [ 0.288090] acpiphp: Slot [19] registered [ 0.289119] acpiphp: Slot [20] registered [ 0.291094] acpiphp: Slot [21] registered [ 0.292107] acpiphp: Slot [22] registered [ 0.294090] acpiphp: Slot [23] registered [ 0.295094] acpiphp: Slot [24] registered [ 0.297129] acpiphp: Slot [25] registered [ 0.298235] acpiphp: Slot [26] registered [ 0.300102] acpiphp: Slot [27] registered [ 0.301113] acpiphp: Slot [28] registered [ 0.303084] acpiphp: Slot [29] registered [ 0.304094] acpiphp: Slot [30] registered [ 0.306123] acpiphp: Slot [31] registered [ 0.307045] PCI host bridge to bus 0000:00 [ 0.308020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.311022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.314024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.320026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.323048] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325673] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.328151] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.336513] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.340046] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.344014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.346020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.348426] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.350633] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.356561] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.375020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.379000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.387930] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.394016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.401016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.419023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.441070] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.453015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.465019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.474963] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.481018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.487017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.501019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.513218] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.524015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.532016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.549020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.562525] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.567013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.572016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.583018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.594000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.597014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.602016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.613022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.624480] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.627365] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.629368] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.632420] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.635253] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.639099] iommu: Default domain type: Passthrough [ 0.641502] SCSI subsystem initialized [ 0.643148] ACPI: bus type USB registered [ 0.645162] usbcore: registered new interface driver usbfs [ 0.648093] usbcore: registered new interface driver hub [ 0.650073] usbcore: registered new device driver usb [ 0.652138] pps_core: LinuxPPS API ver. 1 registered [ 0.654017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.658063] PTP clock support registered [ 0.660145] EDAC MC: Ver: 3.0.0 [ 0.662094] PCI: Using ACPI for IRQ routing [ 0.665064] NetLabel: Initializing [ 0.666010] NetLabel: domain hash size = 128 [ 0.668011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.670084] NetLabel: unlabeled traffic allowed by default [ 0.672093] vgaarb: loaded [ 0.673259] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.675016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.684005] clocksource: Switched to clocksource kvm-clock [ 0.802736] VFS: Disk quotas dquot_6.6.0 [ 0.804354] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.807183] *** VALIDATE ramfs *** [ 0.808605] *** VALIDATE hugetlbfs *** [ 0.810323] pnp: PnP ACPI init [ 0.812842] pnp: PnP ACPI: found 6 devices [ 0.834339] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.838223] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.840299] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.842707] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.845269] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.848211] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.851646] NET: Registered protocol family 2 [ 0.854228] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.859885] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.864383] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.870952] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.875293] TCP: Hash tables configured (established 65536 bind 65536) [ 0.878599] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.882570] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.885832] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.889459] NET: Registered protocol family 1 [ 0.893279] RPC: Registered named UNIX socket transport module. [ 0.895278] RPC: Registered udp transport module. [ 0.896906] RPC: Registered tcp transport module. [ 0.898856] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.901118] NET: Registered protocol family 44 [ 0.902791] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.904356] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.905619] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.907040] PCI: CLS 0 bytes, default 64 [ 0.908168] Unpacking initramfs... [ 2.308393] debug: unmapping init [mem 0xffff9978bcc54000-0xffff9978bffbffff] [ 2.313374] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.316478] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.320330] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.797832] Initialise system trusted keyrings [ 2.799587] Key type blacklist registered [ 2.801419] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.810894] zbud: loaded [ 2.813916] *** VALIDATE nfs *** [ 2.814689] *** VALIDATE nfs4 *** [ 2.816147] pstore: using deflate compression [ 2.819262] Platform Keyring initialized [ 2.911454] NET: Registered protocol family 38 [ 2.913356] Key type asymmetric registered [ 2.914887] Asymmetric key parser 'x509' registered [ 2.916539] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.919680] io scheduler mq-deadline registered [ 2.921438] io scheduler kyber registered [ 2.923335] io scheduler bfq registered [ 2.925256] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.928266] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.930363] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.932473] ACPI: Power Button [PWRF] [ 2.938062] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.944820] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.956429] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.962770] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.975373] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.003229] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.031046] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.036755] Non-volatile memory driver v1.3 [ 3.038957] Linux agpgart interface v0.103 [ 3.068529] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.070625] vda: detected capacity change from 0 to 74698752 [ 3.085088] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.087538] vdb: detected capacity change from 0 to 1073741824 [ 3.101739] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.104041] vdc: detected capacity change from 0 to 2621440000 [ 3.119036] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.122263] vdd: detected capacity change from 0 to 2621440000 [ 3.139792] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.143057] vde: detected capacity change from 0 to 4294967296 [ 3.163783] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.166950] vdf: detected capacity change from 0 to 4294967296 [ 3.175071] libphy: Fixed MDIO Bus: probed [ 3.180347] usbcore: registered new interface driver usbserial_generic [ 3.182368] usbserial: USB Serial support registered for generic [ 3.184876] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.189325] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.191093] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.192857] mousedev: PS/2 mouse device common for all mice [ 3.194824] rtc_cmos 00:05: RTC can wake from S4 [ 3.196915] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.197044] rtc_cmos 00:05: registered as rtc0 [ 3.200707] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.201816] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.204641] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.207285] intel_pstate: CPU model not supported [ 3.213173] hid: raw HID events driver (C) Jiri Kosina [ 3.215789] usbcore: registered new interface driver usbhid [ 3.218576] usbhid: USB HID core driver [ 3.220798] drop_monitor: Initializing network drop monitor service [ 3.223978] Initializing XFRM netlink socket [ 3.226507] NET: Registered protocol family 10 [ 3.230460] Segment Routing with IPv6 [ 3.232427] NET: Registered protocol family 17 [ 3.234960] mpls_gso: MPLS GSO support [ 3.242889] RAS: Correctable Errors collector initialized. [ 3.244909] AVX version of gcm_enc/dec engaged. [ 3.246194] AES CTR mode by8 optimization enabled [ 3.326563] sched_clock: Marking stable (3326541301, 0)->(4175014948, -848473647) [ 3.330400] registered taskstats version 1 [ 3.332721] Loading compiled-in X.509 certificates [ 3.334690] zswap: loaded using pool lzo/zbud [ 3.357565] Key type big_key registered [ 3.372648] Key type encrypted registered [ 3.374077] ima: No TPM chip found, activating TPM-bypass! [ 3.375903] ima: Allocated hash algorithm: sha1 [ 3.377349] ima: No architecture policies found [ 3.378913] evm: Initialising EVM extended attributes: [ 3.380470] evm: security.selinux [ 3.381578] evm: security.ima [ 3.382411] evm: security.capability [ 3.383548] evm: HMAC attrs: 0x1 [ 3.386029] rtc_cmos 00:05: setting system clock to 2026-08-15 16:58:48 UTC (1786813128) [ 3.390961] debug: unmapping init [mem 0xffffffff8d003000-0xffffffff8d1fffff] [ 3.394025] debug: unmapping init [mem 0xffffffff8bd82000-0xffffffff8c058fff] [ 3.404245] Write protecting the kernel read-only data: 28672k [ 3.407138] debug: unmapping init [mem 0xffffffff8a403000-0xffffffff8a5fffff] [ 3.410395] debug: unmapping init [mem 0xffffffff8ad14000-0xffffffff8adfffff] [ 3.443536] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.452254] systemd[1]: Detected virtualization kvm. [ 3.453804] systemd[1]: Detected architecture x86-64. [ 3.455574] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.481845] systemd[1]: No hostname configured. [ 3.483568] systemd[1]: Set hostname to . [ 3.485510] random: systemd: uninitialized urandom read (16 bytes read) [ 3.488036] systemd[1]: Initializing machine ID from random generator. [ 3.528761] random: ln: uninitialized urandom read (6 bytes read) [ 3.610370] random: systemd: uninitialized urandom read (16 bytes read) [ 3.613148] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.618094] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.624533] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.204407] device-mapper: uevent: version 1.0.3 [ 4.206741] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] [ 4.544430] random: fast init done Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.931800] virtio_net virtio0 ens2: renamed from eth0 [ 4.968098] scsi host0: ata_piix [ 4.982419] scsi host1: ata_piix [ 4.984545] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.988072] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.617457] dracut-initqueue[592]: RTNETLINK answers: File exists [ 9.892568] random: crng init done [ 9.893827] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.395110] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.620741] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.877537] SELinux: Disabled at runtime. [ 11.941266] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.949566] systemd[1]: Detected virtualization kvm. [ 11.951370] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.423270] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.426222] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.431744] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.440339] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.443409] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.451606] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.459074] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ 12.483280] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. Mounting Kernel Debug File System... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ 12.959827] squashfs: version 4.0 (2009/01/31) Phillip Lougher Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Mounted /mnt. [ OK ] Started udev Kernel Device Manager. [ 13.310198] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.454280] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.516581] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.537768] EDAC sbridge: Ver: 1.1.2 [ 15.045858] Key type dns_resolver registered [ 15.360204] NFS: Registering the id_resolver key type [ 15.362274] Key type id_resolver registered [ 15.363712] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-server login: [ 45.822059] hrtimer: interrupt took 6440680 ns [ 73.593104] libcfs: loading out-of-tree module taints kernel. [ 73.671277] Key type ._llcrypt registered [ 73.677202] Key type .llcrypt registered [ 73.853807] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 92.204194] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 94.068767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 94.085820] alg: No test for adler32 (adler32-zlib) [ 95.582608] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 96.771129] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 98.559214] Key type lgssc registered [ 100.635770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 118.967213] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.722649] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 178.616702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 178.657812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.017803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 180.063396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 180.157870] Lustre: lustre-MDT0000: new disk, initializing [ 180.312200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.341212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.747603] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.918661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.035899] Lustre: 6531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 200.066423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 200.074033] Lustre: Skipped 1 previous similar message [ 200.216903] Lustre: lustre-MDT0001: new disk, initializing [ 200.370355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.438444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 200.454652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 205.013418] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.742258] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 219.607581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.011031] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 220.012801] Lustre: lustre-OST0000: new disk, initializing [ 220.028768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.043394] Lustre: 8471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.167960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.299866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.302555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.363552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 227.470607] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.765323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.931445] Lustre: lustre-OST0001: new disk, initializing [ 242.938230] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 242.946845] Lustre: 9542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.999683] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 248.400758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 248.422383] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 248.504947] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 251.900322] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.223374] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 273.099711] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 280.076367] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.937584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing yml_node [ 291.277973] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 294.098240] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 296.685846] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 298.589322] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 13:03:42 EDT 2026 [ 319.911559] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 321.448928] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.924924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 338.939847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.958642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.959989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.979900] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.006425] Lustre: Skipped 2 previous similar messages [ 344.841604] Lustre: server umount lustre-MDT0000 complete [ 349.111855] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813474 with bad export cookie 10478095064762515222 [ 349.118830] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 349.124851] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 351.200941] LustreError: 10370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.206573] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.217967] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.217977] Lustre: Skipped 3 previous similar messages [ 351.223363] LustreError: 10370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 351.242325] Lustre: Skipped 1 previous similar message [ 354.215330] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 354.239153] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 355.842252] Lustre: server umount lustre-MDT0001 complete [ 359.804709] Lustre: server umount lustre-OST0000 complete [ 363.142305] Lustre: server umount lustre-OST0001 complete [ 371.621897] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 382.873792] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 424.551654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.891241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 424.916258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 425.127845] Lustre: lustre-MDT0000: new disk, initializing [ 425.255676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.268437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 432.138966] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 445.173688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.312928] Lustre: 15097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 445.388880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 445.392856] Lustre: Skipped 1 previous similar message [ 445.460288] Lustre: lustre-MDT0001: new disk, initializing [ 445.605167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 445.647996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 445.664910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 450.846874] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 461.394857] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 468.985035] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 477.058695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 477.367293] Lustre: lustre-OST0000: new disk, initializing [ 477.369680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 477.378455] Lustre: 17067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 477.463580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.736810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 478.755234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 478.954348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 484.861866] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 494.195533] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 498.914762] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 499.156433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 504.293909] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 504.477174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 509.922232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 509.932166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.935550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 509.950164] Lustre: Skipped 1 previous similar message [ 509.967344] Lustre: Skipped 4 previous similar messages [ 512.872738] Lustre: server umount lustre-OST0000 complete [ 520.163650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.186362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 520.193989] Lustre: Skipped 1 previous similar message [ 520.215528] Lustre: Skipped 1 previous similar message [ 524.583849] Lustre: server umount lustre-MDT0000 complete [ 525.280691] LustreError: 15980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.299468] LustreError: 15980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 528.678836] LustreError: 15091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813653 with bad export cookie 10478095064762516755 [ 528.681540] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.707389] LustreError: 15091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 529.185928] Lustre: server umount lustre-MDT0001 complete [ 535.875520] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 13:07:39 (1786813659) [ 556.964312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.052818] LustreError: 19943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.224851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.685229] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.601700] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 573.920408] LustreError: 19943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.941509] LustreError: 19943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 574.231978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.022480] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 585.951031] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 591.529287] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 593.869948] Lustre: Failing over lustre-MDT0000 [ 594.202217] Lustre: server umount lustre-MDT0000 complete [ 594.405885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.410627] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.418518] Lustre: Skipped 1 previous similar message [ 594.443291] LustreError: 19944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 610.784342] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786813719/real 1786813719] req@ffff997806dd2300 x1873609460188544/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786813735 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 610.814030] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.574483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.352236] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 625.633120] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 625.656303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 625.674447] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 626.184919] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 636.502855] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 640.995248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.004233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 641.004752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 641.012613] Lustre: Skipped 1 previous similar message [ 655.327164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 655.698488] Lustre: server umount lustre-MDT0000 complete [ 659.428157] LustreError: 19939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.451756] LustreError: 19939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 659.594765] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813784 with bad export cookie 10478095064762518169 [ 659.605540] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.608641] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 659.986850] Lustre: server umount lustre-MDT0001 complete [ 669.028817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.763500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.774358] Lustre: Skipped 2 previous similar messages [ 674.422568] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 682.778474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.362462] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 695.652193] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 701.318884] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 703.306927] Lustre: Failing over lustre-MDT0000 [ 703.329274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.335463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 703.354151] Lustre: Skipped 1 previous similar message [ 703.362145] Lustre: Skipped 7 previous similar messages [ 703.674612] Lustre: server umount lustre-MDT0000 complete [ 721.934588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.078489] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.357799] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 726.720528] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 727.521432] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 727.524787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 727.551600] Lustre: Skipped 1 previous similar message [ 727.597806] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 737.067389] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 742.886853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 747.609884] Lustre: server umount lustre-MDT0000 complete [ 748.003136] LustreError: 23284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 748.022132] LustreError: 23284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 751.470270] LustreError: 23263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786813876 with bad export cookie 10478095064762519107 [ 751.476633] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 751.490035] LustreError: 23263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 751.732341] Lustre: server umount lustre-MDT0001 complete [ 760.478885] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 13:11:24 (1786813884) [ 775.918493] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 785.027607] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 823.180965] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 833.312311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 833.552782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 833.578226] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 833.639928] Lustre: lustre-MDT0000: new disk, initializing [ 833.695303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 833.700080] Lustre: Skipped 2 previous similar messages [ 833.709700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 838.690303] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 849.557770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.712758] Lustre: 29657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 850.121346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 854.553327] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 863.468727] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 868.496249] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 875.137353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 875.390231] Lustre: 31629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 876.659269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 876.681516] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 876.727711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 876.738098] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 881.299906] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 889.324578] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 894.648763] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 917.987929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 917.990071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 917.997826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 918.007025] Lustre: Skipped 3 previous similar messages [ 918.013453] Lustre: Skipped 3 previous similar messages [ 919.813138] Lustre: server umount lustre-OST0000 complete [ 926.688903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 926.698527] LustreError: Skipped 1 previous similar message [ 930.000575] Lustre: server umount lustre-MDT0000 complete [ 933.301500] LustreError: 29650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814058 with bad export cookie 10478095064762519513 [ 933.309750] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 933.741300] Lustre: server umount lustre-MDT0001 complete [ 943.119126] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 945.708595] Key type lgssc unregistered [ 945.998891] LNet: 33610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.006611] LNetError: 33610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.028272] LNet: Removed LNI 192.168.201.118@tcp [ 946.800191] Key type .llcrypt unregistered [ 946.805542] Key type ._llcrypt unregistered [ 967.514852] Key type ._llcrypt registered [ 967.516527] Key type .llcrypt registered [ 967.590166] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 968.673880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 968.699018] alg: No test for adler32 (adler32-zlib) [ 969.660430] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 969.883901] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 971.511184] Key type lgssc registered [ 972.456540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 978.805858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 978.844462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.341793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 984.173127] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 990.691705] LustreError: 35015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.726485] LustreError: 35015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 991.701799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.008445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 995.910863] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1001.999919] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1006.646888] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1012.625583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1012.911111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.005131] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1017.581087] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1023.898799] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1025.218779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1043.180506] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 13:16:06 (1786814166) [ 1045.217617] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1047.532304] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1054.660618] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 13:16:18 (1786814178) [ 1063.259476] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 13:16:27 (1786814187) [ 1064.135174] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1071.783700] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1071.790129] Lustre: Skipped 2 previous similar messages [ 1073.654631] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 13:16:37 (1786814197) [ 1077.612508] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1077.617213] Lustre: Skipped 3 previous similar messages [ 1098.893952] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1098.902695] Lustre: Skipped 6 previous similar messages [ 1102.686312] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 13:17:06 (1786814226) [ 1105.983152] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1110.332551] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1118.571767] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 13:17:22 (1786814242) [ 1120.528899] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1120.534663] Lustre: Skipped 3 previous similar messages [ 1133.631786] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 13:17:37 (1786814257) [ 1150.443051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1150.457314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.477640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1151.457240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1151.459678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.449267] Lustre: server umount lustre-OST0000 complete [ 1161.186800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1161.190657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.224099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.231886] Lustre: Skipped 1 previous similar message [ 1166.822436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1166.825166] Lustre: Skipped 1 previous similar message [ 1167.442580] Lustre: server umount lustre-MDT0000 complete [ 1171.151263] LustreError: 34995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814296 with bad export cookie 3224338269427766271 [ 1171.158058] LustreError: 34995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1171.162173] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1171.419946] Lustre: server umount lustre-MDT0001 complete [ 1183.128818] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1186.201853] Key type lgssc unregistered [ 1186.597819] LNet: 42877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.603620] LNetError: 42877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.619359] LNet: Removed LNI 192.168.201.118@tcp [ 1187.709836] Key type .llcrypt unregistered [ 1187.715955] Key type ._llcrypt unregistered [ 1209.593084] Key type ._llcrypt registered [ 1209.605159] Key type .llcrypt registered [ 1209.808808] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 1228.847685] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1229.613025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1229.689523] alg: No test for adler32 (adler32-zlib) [ 1231.024616] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 1231.248076] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1232.935512] Key type lgssc registered [ 1234.248125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1269.230703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1269.245067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.779315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1270.818706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1270.969729] Lustre: lustre-MDT0000: new disk, initializing [ 1271.108656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.163527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1278.244410] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1291.231240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.438694] Lustre: 46029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1291.515065] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1291.521056] Lustre: Skipped 1 previous similar message [ 1291.608656] Lustre: lustre-MDT0001: new disk, initializing [ 1291.728675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1291.769207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1291.780116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1295.873512] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1307.502091] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1315.990580] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1324.865318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1325.309395] Lustre: lustre-OST0000: new disk, initializing [ 1325.315502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1325.319228] Lustre: 48001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1325.450258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.797295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1331.819939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1332.034728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1334.230314] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1345.948417] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1351.979802] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1352.227764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1358.049925] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1358.219913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1362.913271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1362.919966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.922550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1362.928446] Lustre: Skipped 1 previous similar message [ 1368.032614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1368.049937] Lustre: Skipped 2 previous similar messages [ 1373.156099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1373.167458] Lustre: Skipped 1 previous similar message [ 1374.175141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1374.502742] Lustre: server umount lustre-OST0000 complete [ 1383.394559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.398776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1383.432675] Lustre: Skipped 1 previous similar message [ 1383.454407] Lustre: Skipped 1 previous similar message [ 1385.560809] Lustre: server umount lustre-MDT0000 complete [ 1391.933768] LustreError: 46023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814517 with bad export cookie 3498174100786102102 [ 1391.935652] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1391.962373] LustreError: 46023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1392.419867] Lustre: server umount lustre-MDT0001 complete [ 1407.162167] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 13:22:09 (1786814529) [ 1415.820278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.394370] LustreError: 50463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1416.417585] LustreError: 50463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1416.466239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.716533] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1421.793916] LustreError: 50464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.900121] LustreError: 50463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.013917] LustreError: 50464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.257155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.580087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1437.829631] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1446.183430] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1451.582754] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1459.589205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1460.260936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.114723] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1480.026496] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1485.037822] LustreError: 52995:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1486.261397] Lustre: Modifying parameter general.timeout=1 in log params [ 1487.441475] Lustre: Modifying parameter general.timeout=2 in log params [ 1490.205085] Lustre: Modifying parameter general.timeout=4 in log params [ 1490.209429] Lustre: Skipped 1 previous similar message [ 1494.223826] Lustre: Modifying parameter general.timeout=8 in log params [ 1494.226509] Lustre: Skipped 3 previous similar messages [ 1502.341717] Lustre: Modifying parameter general.timeout=15 in log params [ 1502.345346] Lustre: Skipped 6 previous similar messages [ 1518.393653] Lustre: Modifying parameter general.timeout=32 in log params [ 1518.397455] Lustre: Skipped 16 previous similar messages [ 1551.361552] Lustre: Modifying parameter general.timeout=61 in log params [ 1551.370659] Lustre: Skipped 28 previous similar messages [ 1616.326605] Lustre: Modifying parameter general.timeout=125 in log params [ 1616.329991] Lustre: Skipped 63 previous similar messages [ 1633.214989] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 13:25:57 (1786814757) [ 1636.771794] Lustre: 59958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1638.630647] Lustre: 60066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1640.910637] Lustre: 60166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1645.077910] Lustre: 60363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1645.084938] Lustre: 60363:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1678.206388] Lustre: 61844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1678.226408] Lustre: 61844:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1678.253267] LustreError: 61844:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1682.586745] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 13:26:46 (1786814806) [ 1696.743865] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.759190] Lustre: Skipped 1 previous similar message [ 1696.759417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1696.777019] Lustre: Skipped 2 previous similar messages [ 1702.678934] Lustre: server umount lustre-MDT0000 complete [ 1706.977670] LustreError: 50463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.995483] LustreError: 50463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1707.337729] LustreError: 50443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814832 with bad export cookie 3498174100786102984 [ 1707.353694] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1707.357712] LustreError: 50443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1707.921931] Lustre: server umount lustre-MDT0001 complete [ 1722.778069] Lustre: server umount lustre-OST0000 complete [ 1752.877631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.257791] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1753.342097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1753.352417] Lustre: Skipped 27 previous similar messages [ 1753.664294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.934499] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1768.566445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.626622] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1768.647184] Lustre: 64376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1773.425344] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1782.013288] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1789.264112] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1797.656699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1797.879651] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1798.347500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1798.359727] Lustre: Skipped 1 previous similar message [ 1806.539169] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1815.557079] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1817.842525] Lustre: 66942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1830.885964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.886135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1830.896588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1830.896699] Lustre: Skipped 3 previous similar messages [ 1830.901277] Lustre: Skipped 2 previous similar messages [ 1830.913444] LustreError: Skipped 2 previous similar messages [ 1833.148692] Lustre: server umount lustre-OST0000 complete [ 1840.608386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1840.632987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1844.279308] Lustre: server umount lustre-MDT0000 complete [ 1846.267611] LustreError: 64387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1846.293926] LustreError: 64387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1848.805218] LustreError: 64369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786814973 with bad export cookie 3498174100786106190 [ 1848.833617] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1849.199315] Lustre: server umount lustre-MDT0001 complete [ 1864.142830] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1867.401839] Key type lgssc unregistered [ 1867.733414] LNet: 68152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.749198] LNetError: 68152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.776603] LNet: Removed LNI 192.168.201.118@tcp [ 1868.673204] Key type .llcrypt unregistered [ 1868.677944] Key type ._llcrypt unregistered [ 1900.757767] Key type ._llcrypt registered [ 1900.763198] Key type .llcrypt registered [ 1900.899235] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1902.181982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1902.228455] alg: No test for adler32 (adler32-zlib) [ 1903.543338] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 1903.776874] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1905.479199] Key type lgssc registered [ 1906.785809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1915.987870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1916.011623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.587032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.884241] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1927.136325] LustreError: 69569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1927.147453] LustreError: 69569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1932.264266] LustreError: 69570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1932.339922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.033401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1938.004886] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1945.301269] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1951.447242] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1960.990526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1961.744624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.764818] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1978.155754] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1987.129179] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1999.331725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1999.358395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.394792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.916710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.920684] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.928637] Lustre: Skipped 1 previous similar message [ 2002.984252] Lustre: Skipped 1 previous similar message [ 2005.015537] Lustre: server umount lustre-MDT0000 complete [ 2008.033333] LustreError: 69566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.051240] LustreError: 69566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2008.756149] LustreError: 69551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815133 with bad export cookie 9520638941043568045 [ 2008.759453] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2009.188135] Lustre: server umount lustre-MDT0001 complete [ 2023.691902] Lustre: server umount lustre-OST0000 complete [ 2053.259352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.527793] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2053.570496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2053.935967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.801772] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2069.561414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.653570] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2069.701740] Lustre: 74441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2070.333496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2074.674840] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2082.332839] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2089.765132] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2097.860488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2098.087505] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2098.495919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2106.056531] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2116.165118] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2118.165756] Lustre: 77017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2120.647784] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2128.873816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2128.879849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2128.891132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2128.891144] Lustre: Skipped 1 previous similar message [ 2128.893714] LustreError: Skipped 1 previous similar message [ 2128.951397] Lustre: Skipped 1 previous similar message [ 2133.024834] Lustre: server umount lustre-OST0000 complete [ 2139.104076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.106584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.145698] Lustre: Skipped 1 previous similar message [ 2139.162341] Lustre: Skipped 2 previous similar messages [ 2143.705872] Lustre: server umount lustre-MDT0000 complete [ 2144.225551] LustreError: 74454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.246237] LustreError: 74454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2148.213187] LustreError: 75200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815273 with bad export cookie 9520638941043568899 [ 2148.215147] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2148.226349] LustreError: 75200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2148.705620] Lustre: server umount lustre-MDT0001 complete [ 2159.171741] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2162.654350] Key type lgssc unregistered [ 2163.127515] LNet: 78233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.137191] LNetError: 78233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2163.147916] LNet: Removed LNI 192.168.201.118@tcp [ 2164.016127] Key type .llcrypt unregistered [ 2164.017618] Key type ._llcrypt unregistered [ 2191.276527] Key type ._llcrypt registered [ 2191.279207] Key type .llcrypt registered [ 2191.471523] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2192.922569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2193.067275] alg: No test for adler32 (adler32-zlib) [ 2194.447241] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2194.978430] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2196.839323] Key type lgssc registered [ 2198.270300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2206.972260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2206.991911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.378115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2213.948248] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2217.967212] LustreError: 79647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.992968] LustreError: 79647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2223.072222] LustreError: 79646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.132195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.934239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2229.056339] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2238.239893] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2244.427473] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2252.041029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2252.701898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.722244] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2267.708764] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2275.005465] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2276.130182] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2277.461810] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2279.865431] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2279.871843] Lustre: Skipped 1 previous similar message [ 2284.479991] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2284.484765] Lustre: Skipped 4 previous similar messages [ 2293.219911] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2293.224957] Lustre: Skipped 7 previous similar messages [ 2310.057367] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2310.064806] Lustre: Skipped 15 previous similar messages [ 2342.265976] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2342.270128] Lustre: Skipped 30 previous similar messages [ 2390.876654] Lustre: 87755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2394.602329] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 13:38:37 (1786815517) [ 2396.594371] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2399.009490] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:38:42 (1786815522) [ 2406.369757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2406.373515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.379818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2406.414604] Lustre: Skipped 1 previous similar message [ 2406.434709] Lustre: Skipped 1 previous similar message [ 2409.850799] Lustre: server umount lustre-OST0000 complete [ 2416.613974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.617352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2416.631482] Lustre: Skipped 1 previous similar message [ 2416.636837] Lustre: Skipped 1 previous similar message [ 2420.753532] Lustre: server umount lustre-MDT0000 complete [ 2421.728233] LustreError: 79647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.745202] LustreError: 79647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2424.760987] LustreError: 79627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815549 with bad export cookie 345087673232445232 [ 2424.762123] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.775076] LustreError: 79627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2425.362258] Lustre: server umount lustre-MDT0001 complete [ 2437.309178] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2439.879705] Key type lgssc unregistered [ 2440.186479] LNet: 89007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.189120] LNetError: 89007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.208640] LNet: Removed LNI 192.168.201.118@tcp [ 2441.193478] Key type .llcrypt unregistered [ 2441.200804] Key type ._llcrypt unregistered [ 2445.396311] Key type ._llcrypt registered [ 2445.398437] Key type .llcrypt registered [ 2445.494079] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2450.292460] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules [ 2451.562075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2451.583403] alg: No test for adler32 (adler32-zlib) [ 2452.717945] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2452.918671] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2454.583217] Key type lgssc registered [ 2455.936415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2457.384596] LustreError: 90211:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2477.137877] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2479.922908] LustreError: 90211:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2488.291850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2488.312741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.876914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.275930] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2499.051795] LustreError: 91238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.085889] LustreError: 91238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2504.160573] LustreError: 91237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2505.127368] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:40:28 (1786815628) [ 2509.281535] LustreError: 91238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.963034] LustreError: 91269:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2509.975322] LustreError: 91269:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2509.989909] LustreError: 91269:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2510.396597] Lustre: server umount lustre-MDT0000 complete [ 2526.062557] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2529.032381] Key type lgssc unregistered [ 2529.383060] LNet: 92767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2529.399143] LNetError: 92767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2530.491156] LNet: Removed LNI 192.168.201.118@tcp [ 2531.304826] Key type .llcrypt unregistered [ 2531.307105] Key type ._llcrypt unregistered [ 2550.270170] Key type ._llcrypt registered [ 2550.271864] Key type .llcrypt registered [ 2550.432416] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2551.526805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.645732] alg: No test for adler32 (adler32-zlib) [ 2552.568866] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 2552.719339] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2554.383566] Key type lgssc registered [ 2555.628910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2564.314586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2564.345389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.083132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.814635] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2576.353162] LustreError: 93985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.367466] LustreError: 93985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2581.488848] LustreError: 93986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.999055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.199394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2588.232948] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2597.316576] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2603.624270] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2610.782459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2611.246646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.434604] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2627.742865] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2679.098755] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:43:23 (1786815803) [ 2680.762187] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2682.628990] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:43:26 (1786815806) [ 2690.016970] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.020378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.043678] Lustre: Skipped 1 previous similar message [ 2690.055874] Lustre: Skipped 2 previous similar messages [ 2693.405270] Lustre: server umount lustre-MDT0000 complete [ 2695.136602] LustreError: 94718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.153920] LustreError: 94718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2698.412602] LustreError: 93967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815823 with bad export cookie 652503763272343347 [ 2698.419236] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2698.428164] LustreError: 93967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2698.874912] Lustre: server umount lustre-MDT0001 complete [ 2714.680560] Lustre: server umount lustre-OST0000 complete [ 2726.596159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.167942] LustreError: 98287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.432993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.318755] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2732.516630] LustreError: 98288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.771028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2740.792837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2745.166529] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2753.462688] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2759.250806] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2773.017303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.223796] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2773.237517] LustreError: 98277:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2773.270205] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2773.275748] LustreError: 100371:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2773.280392] LustreError: 100371:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2773.287559] LustreError: 100371:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2773.337336] Lustre: server umount lustre-OST0000 complete [ 2773.340227] LustreError: 100371:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2782.183115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2782.276485] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2782.281264] LustreError: 98274:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2782.296175] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2782.303513] LustreError: 100838:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2782.308745] LustreError: 100838:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2782.312580] LustreError: 100838:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2782.345717] Lustre: server umount lustre-OST0000 complete [ 2782.347881] LustreError: 100838:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2792.277256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2792.572462] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2792.585672] Lustre: Found index 0 for lustre-OST0000, updating log [ 2792.605489] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2792.742668] Lustre: lustre-OST0000: new disk, initializing [ 2792.755944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2792.765246] Lustre: 101371:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2793.169885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2800.733912] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2804.229302] Lustre: Failing over lustre-OST0000 [ 2804.339211] Lustre: server umount lustre-OST0000 complete [ 2810.212868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.217063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2810.234964] Lustre: Skipped 2 previous similar messages [ 2810.243207] Lustre: Skipped 1 previous similar message [ 2815.457851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.471710] Lustre: Skipped 1 previous similar message [ 2818.708458] Lustre: server umount lustre-MDT0000 complete [ 2820.581897] LustreError: 98288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.596936] LustreError: 98288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2822.138666] LustreError: 100390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786815947 with bad export cookie 652503763272346091 [ 2822.144177] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2822.151877] LustreError: 100390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2822.648578] Lustre: server umount lustre-MDT0001 complete [ 2830.866319] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:45:54 (1786815954) [ 2848.119444] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2858.944956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.680138] LustreError: 103718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2859.895833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.054962] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2875.715515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.642081] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2886.398230] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2894.425995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2895.580269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2895.582306] Lustre: Skipped 1 previous similar message [ 2895.597792] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2901.738484] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2901.742878] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2901.749770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2902.886676] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2904.822788] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:06 [ 2904.845357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2904.859308] Lustre: lustre-OST0000: Recovery over after 0:03, of 1 clients 1 recovered and 0 were evicted. [ 2904.859683] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2904.872896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2913.299162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2913.527465] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2913.735990] Lustre: lustre-OST0001: new disk, initializing [ 2913.741667] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2913.768550] Lustre: 106041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2920.126346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2920.149429] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2920.286677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2920.800385] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2929.541495] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2940.320592] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2946.019312] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.022285] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2946.027246] Lustre: Skipped 1 previous similar message [ 2946.036703] Lustre: Skipped 2 previous similar messages [ 2951.142307] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2951.153446] Lustre: Skipped 2 previous similar messages [ 2959.327150] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2959.553087] Lustre: server umount lustre-MDT0001 complete [ 2959.966467] LustreError: 103714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2959.983396] LustreError: 103714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2970.779377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.967710] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2970.977450] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2971.038336] Lustre: 103706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2971.796960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2971.805015] Lustre: Skipped 1 previous similar message [ 2972.464425] LustreError: 107587:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2972.477693] LustreError: 107587:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2972.492268] LustreError: 103710:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2972.553856] LustreError: 107597:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -17 [ 2972.565734] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2972.565734] [ 2977.174308] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2977.260680] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2987.320217] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:48:31 (1786816111) [ 3004.400050] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3023.628831] Lustre: 109659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3048.182163] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3052.210927] Lustre: 110794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3140.577290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3140.589258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.598910] Lustre: Skipped 1 previous similar message [ 3140.602836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.610235] Lustre: Skipped 4 previous similar messages [ 3144.084630] Lustre: server umount lustre-MDT0000 complete [ 3146.210714] LustreError: 106336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.217893] LustreError: 106336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3147.408342] LustreError: 103699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816272 with bad export cookie 652503763272347092 [ 3147.411645] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.419298] LustreError: 103699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.757563] Lustre: server umount lustre-MDT0001 complete [ 3161.160787] Lustre: server umount lustre-OST0000 complete [ 3176.483360] Lustre: server umount lustre-OST0001 complete [ 3182.271744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3192.178729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3200.032024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3210.030908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3226.633143] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3236.169056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.190177] Lustre: lustre-MDT0000: reset Object Index mappings [ 3236.677128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3241.920855] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3250.730090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.759220] Lustre: lustre-MDT0001: reset Object Index mappings [ 3256.754459] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3260.241165] Lustre: 115888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3266.771338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3273.991923] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3276.866848] LustreError: 116242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.884479] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3276.885477] LustreError: 116242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3281.025731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3283.848243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3290.103958] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3290.113846] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3291.818517] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3300.297737] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3304.205966] Lustre: 117768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3355.772386] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:54:39 (1786816479) [ 3384.291865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3384.301230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.315579] Lustre: Skipped 3 previous similar messages [ 3384.321734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3384.326192] Lustre: Skipped 4 previous similar messages [ 3388.997087] Lustre: server umount lustre-MDT0000 complete [ 3392.637219] LustreError: 115890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786816517 with bad export cookie 652503763272394146 [ 3392.640112] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3392.643937] LustreError: 115890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3397.601723] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3397.611755] Lustre: Skipped 4 previous similar messages [ 3399.163659] Lustre: server umount lustre-MDT0001 complete [ 3409.069679] Lustre: server umount lustre-OST0000 complete [ 3419.364311] Lustre: server umount lustre-OST0001 complete [ 3426.142661] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3436.786958] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3477.985845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.296876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3478.320637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3478.391944] Lustre: lustre-MDT0000: new disk, initializing [ 3478.459558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3484.311377] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3491.159481] Lustre: server umount lustre-MDT0000 complete [ 3501.780053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.431642] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3502.445396] Lustre: 122038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3502.458685] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3502.852620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3502.862557] Lustre: Skipped 4 previous similar messages [ 3508.365761] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3513.955735] Lustre: server umount lustre-MDT0000 complete [ 3523.417763] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:57:27 (1786816647) [ 3525.108761] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3526.773623] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:57:30 (1786816650) [ 3546.149400] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3556.689549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.031427] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3571.301506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.425740] Lustre: 124089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3571.485277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3571.488881] Lustre: Skipped 1 previous similar message [ 3571.565402] Lustre: lustre-MDT0001: new disk, initializing [ 3571.705561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3571.724120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3576.679171] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3581.569393] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3588.210197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3588.447725] Lustre: lustre-OST0000: new disk, initializing [ 3588.451827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3588.457343] Lustre: 125665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3589.753607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3589.763592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3589.951236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3594.654744] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3604.547902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3604.834340] Lustre: lustre-OST0001: new disk, initializing [ 3604.840606] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3604.848921] Lustre: 126539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3606.439245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3606.452185] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3606.552098] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3612.184804] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3622.467593] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3626.697265] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4024.739603] Lustre: 132971:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4024.744919] Lustre: 132971:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4024.749089] Lustre: 132971:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4024.753347] Lustre: 132971:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4024.759888] Lustre: 132971:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4024.764374] Lustre: 132971:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4318.861122] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:10:42 (1786817442) [ 4338.663232] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.687638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.693208] Lustre: Skipped 3 previous similar messages [ 4338.714244] Lustre: Skipped 9 previous similar messages [ 4342.422912] Lustre: server umount lustre-MDT0000 complete [ 4343.799086] LustreError: 127342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4343.833355] LustreError: 127342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4348.127795] LustreError: 124082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786817473 with bad export cookie 652503763272461724 [ 4348.152673] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4348.158508] LustreError: 124082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4348.897558] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.917453] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4348.940498] Lustre: Skipped 1 previous similar message [ 4348.967226] Lustre: Skipped 1 previous similar message [ 4354.921179] Lustre: server umount lustre-MDT0001 complete [ 4365.443936] Lustre: server umount lustre-OST0000 complete [ 4384.225153] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4384.408969] Lustre: server umount lustre-OST0001 complete [ 4392.046590] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4404.047382] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4441.643861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4441.832624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4441.889715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4442.014157] Lustre: lustre-MDT0000: new disk, initializing [ 4442.110225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4442.113316] Lustre: Skipped 4 previous similar messages [ 4442.135507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4446.266089] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4457.086197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.228240] Lustre: 136026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4457.287861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4457.291099] Lustre: Skipped 1 previous similar message [ 4457.418620] Lustre: lustre-MDT0001: new disk, initializing [ 4457.649641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4457.674842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4463.586691] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4473.421770] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4481.069077] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4488.743270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4489.368698] Lustre: lustre-OST0000: new disk, initializing [ 4489.380193] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4489.394923] Lustre: 137995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4491.149274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4491.159229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4491.326127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4496.030987] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4505.593211] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4516.338860] Lustre: lustre-MDD0000: changelog on [ 4519.806028] Lustre: lustre-MDD0001: changelog on [ 6485.707461] Lustre: lustre-MDD0001: changelog off [ 6487.710332] Lustre: 136902:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6488.579713] Lustre: lustre-MDD0000: changelog off [ 6496.235768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.251090] Lustre: Skipped 1 previous similar message [ 6496.253277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6496.279160] Lustre: Skipped 3 previous similar messages [ 6501.348806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6501.368034] Lustre: Skipped 1 previous similar message [ 6501.826893] Lustre: server umount lustre-OST0000 complete [ 6510.563833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6510.569405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.598864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6511.604508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.835284] Lustre: server umount lustre-MDT0000 complete [ 6516.705083] LustreError: 136903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.733506] LustreError: 136903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6518.185394] LustreError: 136017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819643 with bad export cookie 652503763273284511 [ 6518.195237] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6518.202111] LustreError: 136017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6518.738704] Lustre: server umount lustre-MDT0001 complete [ 6530.610707] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6533.717239] Key type lgssc unregistered [ 6534.053708] LNet: 147149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6534.069291] LNetError: 147149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6534.104987] LNet: Removed LNI 192.168.201.118@tcp [ 6534.952168] Key type .llcrypt unregistered [ 6534.955585] Key type ._llcrypt unregistered [ 6558.168142] Key type ._llcrypt registered [ 6558.172663] Key type .llcrypt registered [ 6558.338782] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6571.921796] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6573.722691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6573.805991] alg: No test for adler32 (adler32-zlib) [ 6574.898372] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 6575.239577] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6576.975662] Key type lgssc registered [ 6578.220823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6617.898579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6617.952031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6619.318081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6619.365716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6619.465309] Lustre: lustre-MDT0000: new disk, initializing [ 6619.557161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6619.599972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6625.671177] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6638.324282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6638.478971] Lustre: 150331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6638.554240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6638.561187] Lustre: Skipped 1 previous similar message [ 6638.651530] Lustre: lustre-MDT0001: new disk, initializing [ 6638.742659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6638.797634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6638.812462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6644.469330] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6655.193465] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6661.030260] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6668.435064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6668.882238] Lustre: lustre-OST0000: new disk, initializing [ 6668.889280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6668.905175] Lustre: 152303:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6669.094599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6674.538721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6674.545996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6674.729878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6676.238479] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6686.389740] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6692.110327] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_module obdecho/obdecho [ 6694.156956] LustreError: 153306:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6694.161559] LustreError: 153306:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6695.921711] LustreError: 153402:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6710.755221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6710.757417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6710.759853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6710.789092] Lustre: Skipped 2 previous similar messages [ 6714.316940] Lustre: server umount lustre-MDT0000 complete [ 6717.978855] LustreError: 150324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786819843 with bad export cookie 17327347338181291351 [ 6717.990922] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6717.991051] LustreError: 150324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6718.270755] Lustre: server umount lustre-MDT0001 complete [ 6732.710206] Lustre: server umount lustre-OST0000 complete [ 6754.986691] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6766.840361] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6802.884148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.142179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6803.170315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6803.246100] Lustre: lustre-MDT0000: new disk, initializing [ 6803.310798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6803.330476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6808.009603] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6819.091826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6819.239928] Lustre: 157116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6819.295578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6819.301038] Lustre: Skipped 1 previous similar message [ 6819.430808] Lustre: lustre-MDT0001: new disk, initializing [ 6819.550018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6819.581509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6819.608430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6825.005310] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6835.401326] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6842.086315] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6849.814215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6850.137305] Lustre: lustre-OST0000: new disk, initializing [ 6850.141150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6850.150197] Lustre: 159085:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6850.238279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6856.497463] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6857.790219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6857.801330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6857.857901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6866.383846] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6871.277679] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6871.570693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6875.562809] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6875.772967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6878.698677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6878.700445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.713160] LustreError: Skipped 1 previous similar message [ 6878.714766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6878.714773] Lustre: Skipped 2 previous similar messages [ 6878.752773] Lustre: Skipped 1 previous similar message [ 6883.720966] Lustre: server umount lustre-OST0000 complete [ 6888.942674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6888.946507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6888.952740] Lustre: Skipped 1 previous similar message [ 6888.959820] Lustre: Skipped 1 previous similar message [ 6894.051603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6894.054599] Lustre: Skipped 1 previous similar message [ 6898.145374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6898.150850] Lustre: Skipped 1 previous similar message [ 6902.239207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6902.447559] Lustre: server umount lustre-MDT0000 complete [ 6903.264784] LustreError: 159714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6903.289724] LustreError: 159714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6906.127711] LustreError: 157109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820031 with bad export cookie 17327347338181292317 [ 6906.138092] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6906.657864] Lustre: server umount lustre-MDT0001 complete [ 6910.410693] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:53:54 (1786820034) [ 6912.253149] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6914.138708] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:53:58 (1786820038) [ 6921.640709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.204619] LustreError: 161498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.285202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6927.060713] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6927.332729] LustreError: 161499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.460847] LustreError: 161498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6935.844273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6941.240057] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6950.141680] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6956.902905] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6965.097508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6965.473785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6965.478484] Lustre: Skipped 1 previous similar message [ 6973.852209] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6982.416057] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6988.398798] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 6992.352554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6992.362364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6992.380563] Lustre: Skipped 1 previous similar message [ 6994.366552] Lustre: server umount lustre-MDT0000 complete [ 6997.488263] LustreError: 161495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6997.505992] LustreError: 161495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6998.921655] LustreError: 161481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820124 with bad export cookie 17327347338181293199 [ 6998.930707] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6998.939371] LustreError: 161481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7002.593441] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7002.614780] Lustre: Skipped 1 previous similar message [ 7008.737145] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7008.737360] LustreError: 163341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.743381] Lustre: Skipped 7 previous similar messages [ 7008.756585] LustreError: 163341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7013.343455] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7013.926745] Lustre: server umount lustre-MDT0001 complete [ 7021.764940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7036.377788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7036.611797] LustreError: 165344:0:(llog.c:1655:llog_backup()) MGC192.168.201.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7036.624117] Lustre: 165344:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7039.385602] LustreError: 165349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7039.416617] LustreError: 165349:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7074.272134] LustreError: 165367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.283945] LustreError: 165367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 7097.631418] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7097.644917] Lustre: 165344:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.118@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7138.292865] LustreError: 165350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7138.295166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7138.314267] LustreError: 165350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 7143.396517] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7143.464423] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7152.246789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7164.897583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7174.221940] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7174.640553] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7181.867418] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7183.461872] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7189.458878] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7190.830458] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7210.464454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7210.466675] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7210.481166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7210.481175] Lustre: Skipped 2 previous similar messages [ 7210.532462] Lustre: Skipped 2 previous similar messages [ 7214.715888] Lustre: server umount lustre-MDT0000 complete [ 7219.262978] LustreError: 161479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820344 with bad export cookie 17327347338181299660 [ 7219.264473] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7219.277411] LustreError: 161479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7219.700607] Lustre: server umount lustre-MDT0001 complete [ 7234.740030] Lustre: server umount lustre-OST0000 complete [ 7259.543391] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7269.582994] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7309.782882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7310.016658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7310.044366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7310.119156] Lustre: lustre-MDT0000: new disk, initializing [ 7310.197816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7310.209191] Lustre: Skipped 1 previous similar message [ 7310.229986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7315.786797] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7328.041599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.199813] Lustre: 170454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7328.523697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7333.235592] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7344.297429] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7350.648514] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7359.030886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7359.497799] Lustre: lustre-OST0000: new disk, initializing [ 7359.506685] Lustre: Skipped 1 previous similar message [ 7359.514297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7359.520573] Lustre: Skipped 2 previous similar messages [ 7359.529656] Lustre: 172432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7366.268067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7366.288696] Lustre: Skipped 1 previous similar message [ 7366.302602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7366.452459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7367.619930] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7381.056238] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7387.650589] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7387.932516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7393.707400] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7394.067087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7397.345408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7397.353486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.372510] Lustre: Skipped 1 previous similar message [ 7397.372921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7397.380776] Lustre: Skipped 2 previous similar messages [ 7402.532288] LustreError: 172417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7402.547331] LustreError: 172417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 31 previous similar messages [ 7402.598815] Lustre: server umount lustre-OST0000 complete [ 7413.242844] Lustre: server umount lustre-MDT0000 complete [ 7418.086662] LustreError: 170445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820543 with bad export cookie 17327347338181316488 [ 7418.089517] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7418.096357] LustreError: 170445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7418.609133] Lustre: server umount lustre-MDT0001 complete [ 7430.475239] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:02:34 (1786820554) [ 7438.551885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7445.267839] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7454.150281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.518276] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7468.076261] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7475.605437] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7485.038805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7493.646298] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7503.993700] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7525.877812] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:04:09 (1786820649) [ 7535.587165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7535.594950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7535.597550] LustreError: Skipped 2 previous similar messages [ 7535.599772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7535.599778] Lustre: Skipped 4 previous similar messages [ 7535.662493] Lustre: Skipped 3 previous similar messages [ 7537.275814] Lustre: server umount lustre-OST0000 complete [ 7547.585842] Lustre: server umount lustre-MDT0000 complete [ 7551.539419] LustreError: 174826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820676 with bad export cookie 17327347338181317370 [ 7551.542963] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7551.546209] LustreError: 174826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7551.780876] Lustre: server umount lustre-MDT0001 complete [ 7565.580611] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7567.849418] Key type lgssc unregistered [ 7568.178208] LNet: 178604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7568.186604] LNetError: 178604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7569.255493] LNet: Removed LNI 192.168.201.118@tcp [ 7570.566989] Key type .llcrypt unregistered [ 7570.568760] Key type ._llcrypt unregistered [ 7591.178548] Key type ._llcrypt registered [ 7591.181254] Key type .llcrypt registered [ 7591.297472] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7592.726159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7592.734380] alg: No test for adler32 (adler32-zlib) [ 7593.750271] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 7593.984039] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7595.681883] Key type lgssc registered [ 7597.102569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7605.889842] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7605.909936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7621.535617] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7626.657606] LustreError: 179852:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 7631.777356] LustreError: 179852:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7631.795423] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7631.810585] LustreError: MGC192.168.201.118@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7631.833413] LustreError: 179852:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7631.850713] LustreError: 179852:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7631.860232] LustreError: 179852:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7631.998871] Lustre: server umount lustre-OST0000 complete [ 7632.008172] LustreError: 179852:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7641.576836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7643.077449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7647.543857] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7653.349921] LustreError: 180352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7653.376357] LustreError: 180352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7655.243886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7655.579885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7659.916913] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7667.543775] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7673.133561] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7680.493045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7680.571831] LustreError: 182174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7680.712468] LustreError: 182167:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7680.721632] Lustre: 182167:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.118@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7680.837241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7687.861344] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7695.502634] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7698.404557] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.416971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7703.527820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7703.539982] Lustre: Skipped 3 previous similar messages [ 7708.640281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7712.223131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7712.533301] Lustre: server umount lustre-MDT0000 complete [ 7713.782875] LustreError: 180351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7713.804241] LustreError: 180351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7717.119757] LustreError: 180333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786820842 with bad export cookie 17990799748041216845 [ 7717.147346] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7717.629215] Lustre: server umount lustre-MDT0001 complete [ 7732.957562] Lustre: server umount lustre-OST0000 complete [ 7743.294668] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7756.326918] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7796.001195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7796.300990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7796.337777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7796.430177] Lustre: lustre-MDT0000: new disk, initializing [ 7796.511369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7796.527703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7800.769572] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7811.512519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7811.647337] Lustre: 185697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7811.692229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7811.695845] Lustre: Skipped 1 previous similar message [ 7811.762523] Lustre: lustre-MDT0001: new disk, initializing [ 7811.841066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7811.877612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7811.892343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7816.477780] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7827.046076] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7834.552628] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7842.457832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7842.669827] Lustre: lustre-OST0000: new disk, initializing [ 7842.671975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7842.681917] Lustre: 187668:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7842.762841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7850.111584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7850.128356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7850.287884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7851.021399] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7861.269335] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7867.093826] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7867.397963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7873.312701] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7873.678535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7876.071221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7876.077436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7876.084208] LustreError: Skipped 1 previous similar message [ 7876.087581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7876.109657] Lustre: Skipped 3 previous similar messages [ 7876.135962] Lustre: Skipped 3 previous similar messages [ 7881.203044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7881.212580] Lustre: Skipped 1 previous similar message [ 7890.401109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7890.839716] Lustre: server umount lustre-OST0000 complete [ 7896.544777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7896.555816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7896.560874] Lustre: Skipped 3 previous similar messages [ 7900.978344] Lustre: server umount lustre-MDT0000 complete [ 7901.670103] LustreError: 185705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7901.680590] LustreError: 185705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7904.607372] LustreError: 185690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821029 with bad export cookie 17990799748041217566 [ 7904.610100] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7904.617848] LustreError: 185690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7904.976666] Lustre: server umount lustre-MDT0001 complete [ 7914.044861] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:10:37 (1786821037) [ 7931.690902] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7934.316388] Key type lgssc unregistered [ 7934.536731] LNet: 190617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7934.540918] LNetError: 190617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7935.593256] LNet: Removed LNI 192.168.201.118@tcp [ 7936.380332] Key type .llcrypt unregistered [ 7936.383741] Key type ._llcrypt unregistered [ 7953.610454] Key type ._llcrypt registered [ 7953.615978] Key type .llcrypt registered [ 7953.755766] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7954.812700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7954.970848] alg: No test for adler32 (adler32-zlib) [ 7956.010857] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 7956.228963] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7957.879180] Key type lgssc registered [ 7959.143872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7966.922390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7966.933592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7968.289831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7972.523716] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7978.469484] LustreError: 191832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7978.492147] LustreError: 191832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7980.686969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.990937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7986.109341] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7994.081387] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8000.699353] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8008.407701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8008.687890] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8008.689611] LustreError: 193697:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8008.694587] Lustre: 193697:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.118@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8008.812777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8015.540792] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8023.173972] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8026.290838] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.295866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8026.313845] Lustre: Skipped 2 previous similar messages [ 8026.324543] Lustre: Skipped 2 previous similar messages [ 8031.592787] Lustre: server umount lustre-MDT0000 complete [ 8031.714503] LustreError: 191832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8031.733778] LustreError: 191832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8035.288353] LustreError: 191813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821160 with bad export cookie 184612974507274230 [ 8035.297502] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8036.577069] LustreError: 192562:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000055c75a54 x1873617701859072/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8036.820066] Lustre: server umount lustre-MDT0001 complete [ 8051.552873] Lustre: server umount lustre-OST0000 complete [ 8062.592783] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8076.953177] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8118.497249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8118.806950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8118.842759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8118.943240] Lustre: lustre-MDT0000: new disk, initializing [ 8119.119234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8119.154803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8124.887517] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8134.388228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8134.509766] Lustre: 197223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8134.541366] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8134.543336] Lustre: Skipped 1 previous similar message [ 8134.545636] LustreError: 198007:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8134.549432] LustreError: 198007:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8134.553166] Lustre: 198007:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8134.567442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8134.570748] Lustre: Skipped 1 previous similar message [ 8134.620500] Lustre: lustre-MDT0001: new disk, initializing [ 8134.703086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8134.736069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8134.755365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8139.350145] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8148.215674] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8154.625743] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8163.110553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.539733] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8163.541984] Lustre: Skipped 1 previous similar message [ 8163.544660] LustreError: 199171:0:(llog.c:1696:llog_backup()) MGC192.168.201.118@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8163.562321] LustreError: 199171:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8163.568875] Lustre: 199171:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8163.581482] Lustre: 199171:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8163.616314] Lustre: lustre-OST0000: new disk, initializing [ 8163.620962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8163.629159] Lustre: 199196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8163.765850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.087936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8169.107277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8169.298246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8171.822290] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8183.048236] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8188.130734] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8188.351642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8192.761992] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8192.940376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8195.048166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8195.049776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.056749] LustreError: Skipped 1 previous similar message [ 8195.057943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8195.111759] Lustre: Skipped 1 previous similar message [ 8200.164738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8200.170835] Lustre: Skipped 1 previous similar message [ 8200.647151] Lustre: server umount lustre-OST0000 complete [ 8205.290873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8205.319635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8205.335479] Lustre: Skipped 1 previous similar message [ 8210.405796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8210.423617] Lustre: Skipped 1 previous similar message [ 8211.803398] Lustre: server umount lustre-MDT0000 complete [ 8215.522504] LustreError: 197231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8215.542801] LustreError: 197231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8216.785372] LustreError: 197216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821341 with bad export cookie 184612974507274951 [ 8216.788103] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8216.801166] LustreError: 197216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8217.028711] Lustre: server umount lustre-MDT0001 complete [ 8228.059662] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:15:52 (1786821352) [ 8247.440120] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8259.078217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8259.356600] LustreError: 202323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8259.405434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8264.003280] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8264.673754] LustreError: 202324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8273.719080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.174700] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8282.414479] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8289.218305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8296.274919] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8304.547407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8304.680546] Lustre: lustre-OST0001: new disk, initializing [ 8304.682859] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8304.687658] Lustre: 204629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8304.787183] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8304.789996] Lustre: Skipped 2 previous similar messages [ 8311.411481] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8311.940491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8311.960111] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8312.141242] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8322.585672] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8330.650576] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8332.313751] Lustre: DEBUG MARKER: ADD OST3 [ 8337.650778] loop: module loaded [ 8337.756130] loop0: detected capacity change from 0 to 204800000 [ 8338.158956] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8338.193048] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8338.226321] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8345.182586] loop0: detected capacity change from 0 to 204800000 [ 8345.486489] Lustre: DEBUG MARKER: STOP OST3 [ 8351.885928] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8352.145230] Lustre: lustre-OST0003: new disk, initializing [ 8352.154834] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8352.177798] Lustre: 206654:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8353.666495] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8353.918402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8353.928220] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8353.969393] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8355.681734] LustreError: 206787:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8356.193922] Lustre: Failing over lustre-OST0003 [ 8356.632944] Lustre: server umount lustre-OST0003 complete [ 8358.883373] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8358.894835] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.906420] Lustre: Skipped 1 previous similar message [ 8358.945390] LustreError: 206673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.975638] LustreError: 206673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8360.754906] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8369.479263] Lustre: DEBUG MARKER: START OST3 again [ 8377.339873] LustreError: 203810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8377.378089] LustreError: 203810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8377.756434] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8378.074872] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8378.076607] Lustre: Skipped 1 previous similar message [ 8378.095518] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8379.480181] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8379.921644] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8379.937550] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8384.159661] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8406.603353] Lustre: Failing over lustre-OST0003 [ 8406.670873] Lustre: server umount lustre-OST0003 complete [ 8409.067580] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8409.076705] Lustre: Skipped 2 previous similar messages [ 8414.179537] LustreError: 206673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8414.213804] LustreError: 206673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8424.421474] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8424.427595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8424.442412] Lustre: Skipped 2 previous similar messages [ 8424.466841] Lustre: Skipped 3 previous similar messages [ 8430.056710] Lustre: server umount lustre-MDT0000 complete [ 8433.193242] LustreError: 205400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821558 with bad export cookie 184612974507275833 [ 8433.196105] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8433.206294] LustreError: 205400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8439.721163] Lustre: server umount lustre-MDT0001 complete [ 8444.418856] Lustre: server umount lustre-OST0000 complete [ 8447.682097] Lustre: server umount lustre-OST0001 complete [ 8454.348615] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8461.610636] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8502.681706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8502.933630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8502.959300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8503.018815] Lustre: lustre-MDT0000: new disk, initializing [ 8503.125745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8507.998677] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8518.879259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8518.980555] Lustre: 212095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8519.072444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8519.075173] Lustre: Skipped 1 previous similar message [ 8519.232610] Lustre: lustre-MDT0001: new disk, initializing [ 8519.360913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8519.370188] Lustre: Skipped 1 previous similar message [ 8519.402859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8519.418466] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8524.205852] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8535.181397] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8541.213375] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8548.562274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8548.967534] Lustre: 214065:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8550.738217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8550.879716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8555.852587] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8566.129954] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8570.822591] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8570.984267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8575.976677] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8576.166377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8581.610179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8581.610691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8581.613587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8581.613595] Lustre: Skipped 8 previous similar messages [ 8581.624108] Lustre: Skipped 4 previous similar messages [ 8581.634251] LustreError: Skipped 1 previous similar message [ 8584.638700] Lustre: server umount lustre-OST0000 complete [ 8591.328413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8594.884959] Lustre: server umount lustre-MDT0000 complete [ 8596.961149] LustreError: 213381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8596.976190] LustreError: 213381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8598.379944] LustreError: 212086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821723 with bad export cookie 184612974507278129 [ 8598.385390] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8598.391318] LustreError: 212086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8598.672629] Lustre: server umount lustre-MDT0001 complete [ 8602.388369] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 15:22:06 (1786821726) [ 8619.064581] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8629.086181] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8667.113282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8667.570317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8667.611261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8667.616571] Lustre: Skipped 1 previous similar message [ 8667.747251] Lustre: lustre-MDT0000: new disk, initializing [ 8667.759036] Lustre: Skipped 1 previous similar message [ 8667.968330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8667.979488] Lustre: Skipped 1 previous similar message [ 8673.044470] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8685.255740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8685.410111] Lustre: 218674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8685.835236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8690.504623] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8701.492585] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8708.709557] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8716.276221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8716.622327] Lustre: 220646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8717.880775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8718.040168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8723.195492] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8734.610632] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8739.720509] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8739.933755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8744.977864] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8745.205911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8748.520813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8748.534520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8748.547541] Lustre: Skipped 2 previous similar messages [ 8748.551866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8748.554991] Lustre: Skipped 3 previous similar messages [ 8754.222298] Lustre: server umount lustre-OST0000 complete [ 8765.352698] Lustre: server umount lustre-MDT0000 complete [ 8769.503546] LustreError: 218665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821894 with bad export cookie 184612974507279011 [ 8769.505063] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8769.511744] LustreError: 218665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8769.959641] Lustre: server umount lustre-MDT0001 complete [ 8779.112108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.632148] LustreError: 222860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8779.705947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8779.714536] Lustre: Skipped 4 previous similar messages [ 8784.789200] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8793.784890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8798.899474] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8806.636431] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8812.504530] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8819.728280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8826.857394] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8834.209344] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8844.770595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8844.784795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8844.800039] Lustre: Skipped 3 previous similar messages [ 8844.822822] LustreError: Skipped 1 previous similar message [ 8844.823676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8844.848911] Lustre: Skipped 6 previous similar messages [ 8846.679529] Lustre: server umount lustre-OST0000 complete [ 8856.620665] Lustre: server umount lustre-MDT0000 complete [ 8860.035220] LustreError: 222841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786821985 with bad export cookie 184612974507279893 [ 8860.045848] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8860.047766] LustreError: 222841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8860.701312] Lustre: server umount lustre-MDT0001 complete [ 8974.153360] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 8976.986928] Key type lgssc unregistered [ 8977.257795] LNet: 226524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8977.276104] LNetError: 226524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8977.298532] LNet: Removed LNI 192.168.201.118@tcp [ 8978.513210] Key type .llcrypt unregistered [ 8978.517689] Key type ._llcrypt unregistered [ 9004.690264] Key type ._llcrypt registered [ 9004.692586] Key type .llcrypt registered [ 9004.787779] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9018.495244] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9019.676598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9019.755770] alg: No test for adler32 (adler32-zlib) [ 9020.763456] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 9020.984690] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9022.703166] Key type lgssc registered [ 9023.988523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9060.093109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9060.115661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.409193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9061.448840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9061.549740] Lustre: lustre-MDT0000: new disk, initializing [ 9061.655642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9061.694492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9065.699572] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9075.591181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.704962] Lustre: 229873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9075.757146] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9075.762103] Lustre: Skipped 1 previous similar message [ 9075.831321] Lustre: lustre-MDT0001: new disk, initializing [ 9075.929089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9075.967932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9075.981096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9080.741902] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9090.389294] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9096.087736] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9102.475275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9102.758300] Lustre: lustre-OST0000: new disk, initializing [ 9102.761980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9102.767905] Lustre: 231843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9102.808642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9109.363575] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9110.133574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9110.153749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9110.331318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9121.851132] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9129.174382] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9129.615150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9133.846605] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9134.023924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9136.096582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9136.104596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9136.109637] LustreError: Skipped 1 previous similar message [ 9136.125390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9136.137232] Lustre: Skipped 1 previous similar message [ 9136.157800] Lustre: Skipped 1 previous similar message [ 9141.232919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9141.237037] Lustre: Skipped 1 previous similar message [ 9141.620592] Lustre: server umount lustre-OST0000 complete [ 9147.873703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9147.885895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9147.916179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9151.460536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9151.464401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9151.493210] Lustre: Skipped 1 previous similar message [ 9152.813834] Lustre: server umount lustre-MDT0000 complete [ 9157.619315] LustreError: 229864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822282 with bad export cookie 15906264138538074012 [ 9157.628917] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9158.067553] Lustre: server umount lustre-MDT0001 complete [ 9178.032723] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9189.137060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9189.680647] LustreError: 234778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.697714] LustreError: 234778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9189.754353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9194.926881] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9194.982674] LustreError: 234779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9199.076804] LustreError: 234778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9204.203657] LustreError: 234779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9204.390274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9204.825774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9210.110142] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9213.145456] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9219.549881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9219.863112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9226.590127] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9236.470659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9236.699729] Lustre: lustre-OST0001: new disk, initializing [ 9236.707847] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9236.716323] Lustre: 237086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9236.837984] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9245.088489] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9245.335890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9245.345434] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9245.510197] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9257.003203] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9266.094151] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9281.847286] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 15:33:25 (1786822405) [ 9286.629214] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9286.635538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9286.646768] Lustre: Skipped 2 previous similar messages [ 9286.648145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9286.657524] Lustre: Skipped 2 previous similar messages [ 9296.873615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9296.893712] Lustre: Skipped 5 previous similar messages [ 9298.401679] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9298.767756] Lustre: server umount lustre-MDT0000 complete [ 9301.987516] LustreError: 236783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9302.017327] LustreError: 236783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9303.233975] LustreError: 234760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822428 with bad export cookie 15906264138538074894 [ 9303.241536] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9303.253886] LustreError: 234760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9303.699736] Lustre: server umount lustre-MDT0001 complete [ 9318.702597] Lustre: server umount lustre-OST0000 complete [ 9333.452825] Lustre: server umount lustre-OST0001 complete [ 9340.919392] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9353.053364] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9394.347729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9394.772678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9394.835124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9394.943096] Lustre: lustre-MDT0000: new disk, initializing [ 9395.053504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9395.089784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9399.838097] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9411.378526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.497829] Lustre: 241035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9411.545082] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9411.548386] Lustre: Skipped 1 previous similar message [ 9411.611230] Lustre: lustre-MDT0001: new disk, initializing [ 9411.724806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9411.736470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9416.807474] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9427.731113] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9434.588187] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9443.124525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9443.560154] Lustre: lustre-OST0000: new disk, initializing [ 9443.566478] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9443.574920] Lustre: 243003:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9445.473487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9445.484095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9445.628402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9450.711298] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9460.532443] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9464.864330] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9465.081587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9470.136869] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9470.305377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9476.584658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9476.585374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9476.596406] LustreError: Skipped 1 previous similar message [ 9476.609992] Lustre: Skipped 1 previous similar message [ 9476.622299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9486.304775] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9486.451499] Lustre: server umount lustre-OST0000 complete [ 9491.937088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9491.953726] Lustre: Skipped 1 previous similar message [ 9496.979756] Lustre: server umount lustre-MDT0000 complete [ 9497.058834] LustreError: 242607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9497.093778] LustreError: 242607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9501.671958] LustreError: 241027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822626 with bad export cookie 15906264138538077029 [ 9501.677043] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9501.695608] LustreError: 241027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9502.164183] Lustre: server umount lustre-MDT0001 complete [ 9510.269817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9510.758038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9510.766643] Lustre: Skipped 2 previous similar messages [ 9514.669249] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9516.004335] LustreError: 245223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9516.025416] LustreError: 245223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9523.766089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9529.159800] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9537.584302] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9544.366498] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9552.588832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9560.042382] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9569.190486] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9718.257434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9718.265864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9718.281914] Lustre: Skipped 1 previous similar message [ 9718.289789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9718.303728] Lustre: Skipped 6 previous similar messages [ 9720.834515] Lustre: server umount lustre-OST0000 complete [ 9731.464932] Lustre: server umount lustre-MDT0000 complete [ 9733.606362] LustreError: 245222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9733.621834] LustreError: 245222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9735.575563] LustreError: 245204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786822860 with bad export cookie 15906264138538077911 [ 9735.583295] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9735.592362] LustreError: 245204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9735.911576] Lustre: server umount lustre-MDT0001 complete [ 9839.976544] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9843.860585] Key type lgssc unregistered [ 9844.465672] LNet: 248830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9844.476419] LNetError: 248830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9844.494902] LNet: Removed LNI 192.168.201.118@tcp [ 9845.433275] Key type .llcrypt unregistered [ 9845.434937] Key type ._llcrypt unregistered [ 9878.664902] Key type ._llcrypt registered [ 9878.667742] Key type .llcrypt registered [ 9878.784510] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9897.150995] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9898.301170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9898.312979] alg: No test for adler32 (adler32-zlib) [ 9899.360589] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [ 9899.559549] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9901.207209] Key type lgssc registered [ 9902.345285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9956.479101] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9969.036385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9969.065255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9970.326555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9970.354979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9970.447507] Lustre: lustre-MDT0000: new disk, initializing [ 9970.552173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9970.563281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9975.477245] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9986.413203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9986.537984] Lustre: 252912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9986.598065] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9986.602316] Lustre: Skipped 1 previous similar message [ 9986.722156] Lustre: lustre-MDT0001: new disk, initializing [ 9986.828899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9986.860093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9986.877194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9991.804892] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9997.221201] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10005.301648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10005.660948] Lustre: lustre-OST0000: new disk, initializing [10005.663344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10005.668245] Lustre: 254543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10005.820792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10006.432804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10006.459770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10006.652697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10013.236430] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10024.446923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10024.636440] Lustre: lustre-OST0001: new disk, initializing [10024.638868] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10024.643262] Lustre: 255418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10024.749014] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10030.270735] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10031.222926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10031.236176] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10031.325149] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10042.159001] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10054.065917] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10067.426555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10067.434575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10067.455419] Lustre: Skipped 1 previous similar message [10071.306606] Lustre: server umount lustre-MDT0000 complete [10075.394431] LustreError: 252905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823200 with bad export cookie 11930486223596036113 [10075.396203] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10075.403742] LustreError: 252905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10075.771734] Lustre: server umount lustre-MDT0001 complete [10089.101604] Lustre: server umount lustre-OST0000 complete [10093.023375] Lustre: 250418:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786823202/real 1786823202] req@ffff99791fad8000 x1873619739295616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786823218 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10093.061203] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10093.085644] Lustre: Skipped 2 previous similar messages [10093.943158] Lustre: server umount lustre-OST0001 complete [10111.389959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10123.361844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10132.242771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10141.695651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10164.560292] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10177.190153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10177.211464] Lustre: lustre-MDT0000: reset Object Index mappings [10177.663896] LustreError: 260430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10177.685892] LustreError: 260430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10177.768127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10183.138532] LustreError: 260431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10183.446584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10187.239777] LustreError: 260431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10192.363496] LustreError: 260430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10195.221889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10195.294864] Lustre: lustre-MDT0001: reset Object Index mappings [10196.252785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10202.496033] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10205.947898] Lustre: 261572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10212.889191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10213.209074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10219.010362] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10221.424461] LustreError: 261926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10222.482454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10229.647729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10235.410548] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10237.756920] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10245.642252] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10250.916788] Lustre: 263445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10257.379082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10257.398298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10257.426201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10257.436577] Lustre: Skipped 3 previous similar messages [10258.634266] Lustre: server umount lustre-MDT0000 complete [10260.964944] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.967273] LustreError: 260430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.004938] LustreError: 260430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10263.091942] LustreError: 260412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823388 with bad export cookie 11930486223596042903 [10263.101648] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10263.725658] Lustre: server umount lustre-MDT0001 complete [10278.881840] Lustre: server umount lustre-OST0000 complete [10282.212356] Lustre: 250419:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786823391/real 1786823391] req@ffff9978038f2d80 x1873619739348352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786823407 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10282.260813] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10282.289856] Lustre: Skipped 2 previous similar messages [10283.073992] Lustre: server umount lustre-OST0001 complete [10303.937907] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:50:27 (1786823427) [10320.265985] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10330.469922] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10365.297703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10365.547163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10365.591667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10365.717884] Lustre: lustre-MDT0000: new disk, initializing [10365.837221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10365.843694] Lustre: Skipped 1 previous similar message [10365.870225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10370.425188] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10382.529977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10382.727586] Lustre: 267699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10382.824504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10382.832141] Lustre: Skipped 1 previous similar message [10382.939613] Lustre: lustre-MDT0001: new disk, initializing [10383.108975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10383.126275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10388.771528] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10400.009203] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10406.053832] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10414.418743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10415.005387] Lustre: lustre-OST0000: new disk, initializing [10415.015608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10415.028948] Lustre: 269669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10416.473359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10416.481684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10416.670783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10422.289232] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10432.774692] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10438.711940] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10438.969519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10444.787124] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10445.161887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10452.460897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10452.468185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10452.484671] LustreError: Skipped 1 previous similar message [10452.489069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10452.542071] Lustre: Skipped 1 previous similar message [10454.194342] Lustre: server umount lustre-OST0000 complete [10460.132687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10460.150750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10460.160188] Lustre: Skipped 1 previous similar message [10467.212750] Lustre: server umount lustre-MDT0000 complete [10467.810439] LustreError: 267712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.420362] LustreError: 267691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823596 with bad export cookie 11930486223596044996 [10471.421843] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10471.428749] LustreError: 267691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10471.732841] Lustre: server umount lustre-MDT0001 complete [10491.203505] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10503.219410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.117434] LustreError: 272603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10504.136706] LustreError: 272603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10504.207694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10504.211707] Lustre: Skipped 2 previous similar messages [10510.647881] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10520.291537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.174584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10530.938806] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10537.820696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10545.424372] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10553.924127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10554.096520] Lustre: lustre-OST0001: new disk, initializing [10554.099893] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10554.106992] Lustre: 274913:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10555.714190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10555.718474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10555.864335] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10561.614323] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10571.697554] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10574.742863] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10580.545745] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10580.573594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10591.719736] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10591.734602] Lustre: Skipped 4 previous similar messages [10591.735887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10591.755893] Lustre: Skipped 5 previous similar messages [10594.830600] Lustre: server umount lustre-MDT0000 complete [10596.853398] LustreError: 272604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10596.882223] LustreError: 272604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [10598.857420] LustreError: 272585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823723 with bad export cookie 11930486223596045878 [10598.867798] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10598.881324] LustreError: 272585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10599.310763] Lustre: server umount lustre-MDT0001 complete [10613.970540] Lustre: server umount lustre-OST0000 complete [10628.382948] Lustre: server umount lustre-OST0001 complete [10649.011575] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10662.809481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10663.430845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10663.440615] Lustre: Skipped 3 previous similar messages [10668.692565] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10677.924816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10683.313467] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10686.121219] Lustre: 279115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10692.585535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10699.196278] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10707.468354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10708.749114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10708.816297] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10714.959126] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10722.953839] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10727.059483] Lustre: 280993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10732.427485] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10732.579252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10745.260181] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:57:48 (1786823868) [10765.287387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10765.308089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10765.342105] Lustre: Skipped 2 previous similar messages [10765.380024] Lustre: Skipped 3 previous similar messages [10768.761743] Lustre: server umount lustre-MDT0000 complete [10770.401612] LustreError: 277973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10770.439724] LustreError: 277973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10772.828864] LustreError: 277954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786823897 with bad export cookie 11930486223596052409 [10772.833199] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10772.848881] LustreError: 277954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10782.691644] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10782.701932] Lustre: Skipped 5 previous similar messages [10787.295297] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10787.656352] Lustre: server umount lustre-MDT0001 complete [10798.329949] Lustre: server umount lustre-OST0000 complete [10802.171556] Lustre: server umount lustre-OST0001 complete [10809.505195] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10820.333560] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10872.307123] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10882.283845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10882.790413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10882.818289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10882.953955] Lustre: lustre-MDT0000: new disk, initializing [10883.109439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10888.405922] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10898.259126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10898.350318] Lustre: 285233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10898.572029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10903.316446] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10914.523049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10914.686691] Lustre: 286866:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10916.045578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10922.486493] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10935.618148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10935.918752] Lustre: 287739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10936.078423] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10936.086685] Lustre: Skipped 6 previous similar messages [10937.920093] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10937.937377] Lustre: Skipped 1 previous similar message [10938.051752] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10943.289785] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10953.185920] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10957.058629] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10957.064405] Lustre: Skipped 1 previous similar message [10972.347951] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 16:01:36 (1786824096) [10975.202514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10975.217087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10975.233050] Lustre: Skipped 4 previous similar messages [10975.240484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10980.676765] Lustre: server umount lustre-MDT0000 complete [10984.333454] LustreError: 285227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824109 with bad export cookie 11930486223596060088 [10984.339401] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10984.342102] LustreError: 285227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10984.838324] Lustre: server umount lustre-MDT0001 complete [10991.031595] Lustre: server umount lustre-OST0000 complete [10995.564703] Lustre: server umount lustre-OST0001 complete [11005.738982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11012.692529] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11022.224390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11027.759104] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11036.589623] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11042.545378] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11049.504576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11050.986233] LustreError: 291823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11051.001507] LustreError: 291823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [11056.144488] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11063.914604] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11085.458052] loop1: detected capacity change from 0 to 204800000 [11085.779052] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11085.805588] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11085.817314] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11091.288299] loop1: detected capacity change from 0 to 204800000 [11097.300852] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11097.400774] LustreError: 289981:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs22092: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11097.419398] LustreError: 289981:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs22092 [11097.423105] LustreError: 289981:0:(mgs_handler.c:517:mgs_target_reg()) fs22092-MDT0001: Error checking target index: rc = -13 [11097.432564] LustreError: fs22092-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11097.437987] LustreError: 293713:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11097.443042] LustreError: 293713:0:(tgt_mount.c:2129:server_put_super()) no obd fs22092-MDT0001 [11097.452392] LustreError: 293713:0:(tgt_mount.c:116:server_deregister_mount()) fs22092-MDT0001 not registered [11097.511667] Lustre: server umount fs22092-MDT0001 complete [11097.516538] LustreError: 293713:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11106.435511] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11106.502537] Lustre: Setting parameter fs22092-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22092-MDT0001 [11106.551296] Lustre: srv-fs22092-MDT0001: No data found on store. Initialize space. [11106.555382] Lustre: Skipped 4 previous similar messages [11106.654701] Lustre: fs22092-MDT0001: new disk, initializing [11106.659486] Lustre: Skipped 3 previous similar messages [11106.829271] Lustre: srv-fs22092-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11106.837094] Lustre: 294179:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22092-MDT0001-mdtlov: can't create new llog: rc = -115 [11106.853613] LustreError: 294179:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22092-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11110.653727] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11116.135560] Lustre: server umount fs22092-MDT0001 complete [11128.813725] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11129.344853] loop1: detected capacity change from 0 to 204800000 [11135.713707] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11136.366640] Lustre: 295747:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22092-MDT0001-mdtlov: can't create new llog: rc = -115 [11136.383497] LustreError: 295747:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22092-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11141.543570] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11146.177665] Lustre: server umount fs22092-MDT0001 complete [11149.289175] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11155.937490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11155.953120] Lustre: Skipped 5 previous similar messages [11155.962916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11155.971059] Lustre: Skipped 6 previous similar messages [11170.271140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11170.472182] Lustre: server umount lustre-MDT0000 complete [11179.235744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11179.361458] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11183.993275] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11185.132286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11193.232491] loop1: detected capacity change from 0 to 204800000 [11202.791821] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11203.563491] Lustre: 298195:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22092-MDT0001-mdtlov: can't create new llog: rc = -115 [11203.579654] LustreError: 298195:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22092-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11208.844647] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11213.698735] Lustre: server umount fs22092-MDT0001 complete [11235.810652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11235.820222] LustreError: Skipped 1 previous similar message [11241.853312] Lustre: server umount lustre-OST0000 complete [11260.383871] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11260.795096] Lustre: server umount lustre-MDT0000 complete [11265.031114] LustreError: 289973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824390 with bad export cookie 11930486223596062769 [11265.036846] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11265.040555] LustreError: 289973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11265.408354] Lustre: server umount lustre-MDT0001 complete [11278.641074] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11281.899122] Key type lgssc unregistered [11282.166770] LNet: 300545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11282.173236] LNetError: 300545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11282.187852] LNet: Removed LNI 192.168.201.118@tcp [11283.128266] Key type .llcrypt unregistered [11283.133099] Key type ._llcrypt unregistered [11308.916743] Key type ._llcrypt registered [11308.924950] Key type .llcrypt registered [11309.055373] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11310.579802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11310.612336] alg: No test for adler32 (adler32-zlib) [11311.733665] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [11312.085230] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11313.807179] Key type lgssc registered [11315.339849] Lustre: Echo OBD driver; http://www.lustre.org/ [11323.306909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11323.350647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11324.845370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11329.072836] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11334.116979] LustreError: 301819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11334.127304] LustreError: 301819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11337.275107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11337.839168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11342.192716] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11349.410442] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11354.960289] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11361.704659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11362.281867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11363.309838] LustreError: 303648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11368.423685] LustreError: 303649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11368.552265] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11374.562791] LustreError: 303647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11374.575784] LustreError: 303647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11376.012521] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11378.639611] LustreError: 303648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11389.029928] LustreError: 303648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11389.038585] LustreError: 303648:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11395.443417] loop1: detected capacity change from 0 to 204800000 [11395.668897] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11395.685279] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11395.694730] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11402.111526] loop1: detected capacity change from 0 to 204800000 [11405.285616] LustreError: 303649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11405.318729] LustreError: 303649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11408.918683] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11409.236543] Lustre: fs22176-OST0000: new disk, initializing [11409.241405] Lustre: srv-fs22176-OST0000: No data found on store. Initialize space. [11409.250805] Lustre: 305551:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11409.402297] Lustre: fs22176-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11418.515822] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11427.771861] Lustre: server umount fs22176-OST0000 complete [11440.228078] LustreError: 303647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.251032] LustreError: 303647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11443.902508] loop1: detected capacity change from 0 to 204800000 [11446.256889] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11450.704091] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11451.024494] Lustre: fs22176-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11451.370366] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11457.257193] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11464.707523] loop2: detected capacity change from 0 to 204800000 [11464.896977] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11464.906339] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11464.924521] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11470.559276] loop2: detected capacity change from 0 to 204800000 [11477.074779] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11477.181058] LustreError: fs22176-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11477.189428] LustreError: fs22176-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11477.192378] LustreError: 308594:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11477.197917] LustreError: 308594:0:(tgt_mount.c:2129:server_put_super()) no obd fs22176-OST0001 [11477.206041] LustreError: 308594:0:(tgt_mount.c:116:server_deregister_mount()) fs22176-OST0001 not registered [11477.238531] Lustre: server umount fs22176-OST0001 complete [11477.243061] LustreError: 308594:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11489.758650] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11490.028624] Lustre: fs22176-OST0001: new disk, initializing [11490.034697] Lustre: srv-fs22176-OST0001: No data found on store. Initialize space. [11490.041275] Lustre: 309146:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11490.132803] Lustre: fs22176-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11496.943430] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11505.878571] Lustre: server umount fs22176-OST0000 complete [11506.778288] LustreError: 303648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.794993] LustreError: 303648:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11515.801185] Lustre: server umount fs22176-OST0001 complete [11528.538736] loop1: detected capacity change from 0 to 204800000 [11534.658441] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11534.987725] Lustre: fs22176-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11542.607348] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11549.593248] loop2: detected capacity change from 0 to 204800000 [11556.101330] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11563.294325] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11572.135752] Lustre: server umount fs22176-OST0000 complete [11581.876588] Lustre: server umount fs22176-OST0001 complete [11594.753234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11594.761765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11594.770845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11594.781089] Lustre: Skipped 1 previous similar message [11599.845734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11599.853459] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11599.855897] Lustre: Skipped 2 previous similar messages [11600.792825] Lustre: server umount lustre-OST0000 complete [11604.963255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11604.973395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11610.081024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11610.091351] Lustre: Skipped 2 previous similar messages [11610.889667] Lustre: server umount lustre-MDT0000 complete [11614.778010] LustreError: 301800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824739 with bad export cookie 16362598366752457369 [11614.782211] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11614.818743] LustreError: 301800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11615.261637] Lustre: server umount lustre-MDT0001 complete [11630.335588] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11633.856522] Key type lgssc unregistered [11634.293213] LNet: 315076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11634.305562] LNetError: 315076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11634.328251] LNet: Removed LNI 192.168.201.118@tcp [11635.635259] Key type .llcrypt unregistered [11635.639384] Key type ._llcrypt unregistered [11668.512777] Key type ._llcrypt registered [11668.520440] Key type .llcrypt registered [11668.613503] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11685.476074] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11686.881810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11687.066127] alg: No test for adler32 (adler32-zlib) [11688.278399] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [11688.514550] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11690.127172] Key type lgssc registered [11691.633107] Lustre: Echo OBD driver; http://www.lustre.org/ [11726.938180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11726.965339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11728.228217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11728.274264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11728.399372] Lustre: lustre-MDT0000: new disk, initializing [11728.504799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11728.531439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11732.702828] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11743.884797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.039353] Lustre: 318937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11744.138393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11744.145811] Lustre: Skipped 1 previous similar message [11744.206742] Lustre: lustre-MDT0001: new disk, initializing [11744.308716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11744.362939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11744.383472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11750.298311] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11759.984985] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11765.733543] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11772.758488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11773.048558] Lustre: lustre-OST0000: new disk, initializing [11773.052236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11773.063344] Lustre: 320904:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11773.196707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11778.683364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11778.692953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11778.847544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11779.745883] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11789.887525] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11792.229255] Lustre: Mounted lustre-client - version 2.17.57_1_g2c0d074 [11793.179901] Lustre: Unmounted lustre-client [11802.597892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11802.602274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11802.608700] Lustre: Skipped 1 previous similar message [11802.616983] Lustre: Skipped 1 previous similar message [11807.731175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11807.739965] Lustre: Skipped 1 previous similar message [11808.046182] Lustre: server umount lustre-OST0000 complete [11812.834087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11812.858436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11817.953113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11817.955785] Lustre: Skipped 1 previous similar message [11818.349456] Lustre: server umount lustre-MDT0000 complete [11821.754220] LustreError: 318928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786824946 with bad export cookie 18417958400737880701 [11821.760090] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11821.766025] LustreError: 318928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11822.281772] Lustre: server umount lustre-MDT0001 complete [11832.819547] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [11835.496871] Key type lgssc unregistered [11844.191874] LNet: 1 local NIs in recovery (showing 1): 192.168.201.118@tcp [11845.027161] LNet: 323070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11845.035788] LNetError: 323070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11924.903464] LNet: Removed LNI 192.168.201.118@tcp [11926.014252] Key type .llcrypt unregistered [11926.016903] Key type ._llcrypt unregistered [11945.055912] Key type ._llcrypt registered [11945.058538] Key type .llcrypt registered [11945.184990] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11965.647705] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11967.567404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11967.609573] alg: No test for adler32 (adler32-zlib) [11968.613917] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [11968.825135] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [11970.479432] Key type lgssc registered [11971.451524] Lustre: Echo OBD driver; http://www.lustre.org/ [12004.269893] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:18:48 (1786825128) [12010.161169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12010.177272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12011.342500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12011.365113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12011.447938] Lustre: lustre-MDT0000: new disk, initializing [12011.634141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12011.673399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12017.344884] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12029.919322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12030.078627] Lustre: 326298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12030.206959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12030.214708] Lustre: Skipped 1 previous similar message [12030.409190] Lustre: lustre-MDT0001: new disk, initializing [12030.654264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12030.709120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12030.727929] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12036.978706] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12047.597169] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12052.982534] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12060.065284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12060.274240] Lustre: lustre-OST0000: new disk, initializing [12060.279089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12060.285593] Lustre: 328268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12060.411840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12066.874193] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12070.527211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12070.536497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12070.706790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12077.748159] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12096.492086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12096.493231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12096.536803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12100.155372] Lustre: server umount lustre-OST0000 complete [12106.728172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12106.738688] Lustre: Skipped 1 previous similar message [12106.746098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12106.751444] Lustre: Skipped 2 previous similar messages [12111.280525] Lustre: server umount lustre-MDT0000 complete [12115.761914] LustreError: 326292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825240 with bad export cookie 7678386289671457693 [12115.770243] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12115.774024] LustreError: 326292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12116.262620] Lustre: server umount lustre-MDT0001 complete [12128.015568] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12130.815753] Key type lgssc unregistered [12131.166685] LNet: 330177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12131.173826] LNetError: 330177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12131.196347] LNet: Removed LNI 192.168.201.118@tcp [12132.044631] Key type .llcrypt unregistered [12132.047204] Key type ._llcrypt unregistered [12151.098330] Key type ._llcrypt registered [12151.100473] Key type .llcrypt registered [12151.211270] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12152.482375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12152.671165] alg: No test for adler32 (adler32-zlib) [12153.705612] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [12154.044706] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12155.768132] Key type lgssc registered [12156.874087] Lustre: Echo OBD driver; http://www.lustre.org/ [12165.105096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12165.125701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12166.743419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12171.742168] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12176.868972] LustreError: 331443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12176.898105] LustreError: 331443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12181.986980] LustreError: 331442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12182.647122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12182.969546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12189.221885] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12199.147887] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12205.001724] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12212.735899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12213.221581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12218.911395] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12226.767618] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12247.126485] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12248.717648] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12254.347643] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [12254.696225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12254.708620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12257.251136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12257.260560] Lustre: Skipped 1 previous similar message [12257.280860] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12258.909650] Lustre: server umount lustre-MDT0000 complete [12259.417429] LustreError: 331438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12262.369788] LustreError: 332373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12262.386394] LustreError: 332373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12263.108545] LustreError: 331424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825388 with bad export cookie 79071083610181554 [12263.138917] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12264.582269] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [12264.605090] Lustre: Skipped 2 previous similar messages [12267.493406] LustreError: 331437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12267.503853] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12267.515662] LustreError: 331437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12267.539717] Lustre: Skipped 1 previous similar message [12267.559892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12269.559298] Lustre: server umount lustre-MDT0001 complete [12273.582123] Lustre: server umount lustre-OST0000 complete [12282.231521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12287.538366] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12290.150589] Lustre: 335648:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.118@tcp [12292.431725] Lustre: server umount MGS complete [12299.699702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12300.065315] LustreError: 336220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12300.087231] LustreError: 336220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12300.136662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12304.054821] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12311.911249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.196652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12316.831780] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12324.850413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12325.316420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12331.009137] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12337.930761] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12339.466422] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12345.560259] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12347.673922] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12352.993814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12352.999693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12353.008551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12353.030037] Lustre: Skipped 1 previous similar message [12353.046171] Lustre: Skipped 2 previous similar messages [12354.775920] Lustre: server umount lustre-OST0000 complete [12363.232249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12363.240288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12363.257959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12363.276826] Lustre: Skipped 1 previous similar message [12365.439689] Lustre: server umount lustre-MDT0000 complete [12368.352121] LustreError: 336221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12368.385317] LustreError: 336221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12369.042104] LustreError: 336200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825494 with bad export cookie 79071083610182394 [12369.048745] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12369.068323] LustreError: 336200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12369.377874] Lustre: server umount lustre-MDT0001 complete [12381.477687] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12384.228461] Key type lgssc unregistered [12384.500617] LNet: 339658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12384.505724] LNetError: 339658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12384.523430] LNet: Removed LNI 192.168.201.118@tcp [12385.262392] Key type .llcrypt unregistered [12385.264232] Key type ._llcrypt unregistered [12403.358386] Key type ._llcrypt registered [12403.361018] Key type .llcrypt registered [12403.509381] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [12421.288472] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12422.876392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12422.900433] alg: No test for adler32 (adler32-zlib) [12423.843218] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [12424.076588] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12425.783231] Key type lgssc registered [12427.020252] Lustre: Echo OBD driver; http://www.lustre.org/ [12465.938250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12465.976692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12467.286558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12467.334724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12467.437619] Lustre: lustre-MDT0000: new disk, initializing [12467.520574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12467.564761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12472.053330] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12481.951090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12482.064730] Lustre: 342821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12482.106632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12482.109321] Lustre: Skipped 1 previous similar message [12482.168480] Lustre: lustre-MDT0001: new disk, initializing [12482.228710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12482.263632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12482.274853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12486.619252] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12495.383202] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12501.390577] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12511.245475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12511.698928] Lustre: lustre-OST0000: new disk, initializing [12511.702606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12511.710878] Lustre: 344790:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12511.866678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12517.469809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12517.500159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12517.672693] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12520.743901] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12530.768545] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12536.019416] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12536.240983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12540.945901] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12541.156530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12543.470106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12543.490853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12543.525415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12543.534804] Lustre: Skipped 1 previous similar message [12548.585694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12548.595827] Lustre: Skipped 1 previous similar message [12549.441603] Lustre: server umount lustre-OST0000 complete [12554.209124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12554.211386] LustreError: Skipped 1 previous similar message [12554.219500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12554.234144] Lustre: Skipped 1 previous similar message [12554.241365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12558.819812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12558.823697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12560.280964] Lustre: server umount lustre-MDT0000 complete [12564.140421] LustreError: 344789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825689 with bad export cookie 13433131498659356030 [12564.156211] LustreError: 344789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12564.156545] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12564.577984] Lustre: server umount lustre-MDT0001 complete [12574.689398] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 16:28:18 (1786825698) [12595.345641] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [12606.337817] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12648.580215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12649.143382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12649.187798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12649.343541] Lustre: lustre-MDT0000: new disk, initializing [12649.496196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12649.534967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12655.873750] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12665.709922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12665.807465] Lustre: 349553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12665.857722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12665.862180] Lustre: Skipped 1 previous similar message [12665.921646] Lustre: lustre-MDT0001: new disk, initializing [12666.009147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12666.052060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12666.064089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12671.583524] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12682.102300] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12689.803788] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12698.165545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12698.370832] Lustre: lustre-OST0000: new disk, initializing [12698.373397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12698.376864] Lustre: 351524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12698.463802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12705.470676] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12707.399466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12707.408964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12707.542320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12715.764850] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12747.564466] Lustre: server umount lustre-OST0000 complete [12753.378082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12753.380887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12753.404168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12753.410518] Lustre: Skipped 1 previous similar message [12761.145651] Lustre: server umount lustre-MDT0000 complete [12764.064133] Lustre: 341043:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786825873/real 1786825873] req@00000000becce1b7 x1873622386959744/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786825889 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12764.094373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12764.102478] Lustre: Skipped 1 previous similar message [12764.106921] LustreError: 349560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12764.131596] LustreError: 349560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12765.335513] LustreError: 349544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786825890 with bad export cookie 13433131498659356912 [12765.339160] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12765.347801] LustreError: 349544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12765.647768] Lustre: server umount lustre-MDT0001 complete [12775.788798] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12778.179432] Key type lgssc unregistered [12778.462686] LNet: 353678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12778.467871] LNetError: 353678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12778.482103] LNet: Removed LNI 192.168.201.118@tcp [12779.346244] Key type .llcrypt unregistered [12779.347191] Key type ._llcrypt unregistered [12859.401458] Key type ._llcrypt registered [12859.403557] Key type .llcrypt registered [12859.487343] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12859.889484] Key type .llcrypt unregistered [12859.891452] Key type ._llcrypt unregistered [12930.850278] Key type ._llcrypt registered [12930.852954] Key type .llcrypt registered [12931.047348] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12931.545858] Key type .llcrypt unregistered [12931.548936] Key type ._llcrypt unregistered [12961.331516] Key type ._llcrypt registered [12961.334483] Key type .llcrypt registered [12961.414348] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12961.778430] Key type .llcrypt unregistered [12961.784887] Key type ._llcrypt unregistered [12988.655235] Key type ._llcrypt registered [12988.657416] Key type .llcrypt registered [12988.757668] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12988.977930] Key type .llcrypt unregistered [12988.980563] Key type ._llcrypt unregistered [13058.888757] Key type ._llcrypt registered [13058.891362] Key type .llcrypt registered [13059.023233] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13059.364178] Key type .llcrypt unregistered [13059.365975] Key type ._llcrypt unregistered [13086.935557] Key type ._llcrypt registered [13086.937867] Key type .llcrypt registered [13087.036226] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13087.341401] Key type .llcrypt unregistered [13087.344863] Key type ._llcrypt unregistered [13115.164263] Key type ._llcrypt registered [13115.166427] Key type .llcrypt registered [13115.313841] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13115.700388] Key type .llcrypt unregistered [13115.702075] Key type ._llcrypt unregistered [13145.228421] Key type ._llcrypt registered [13145.230383] Key type .llcrypt registered [13145.396919] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13145.780662] Key type .llcrypt unregistered [13145.782972] Key type ._llcrypt unregistered [13176.307216] Key type ._llcrypt registered [13176.309404] Key type .llcrypt registered [13176.407601] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13176.724504] Key type .llcrypt unregistered [13176.727162] Key type ._llcrypt unregistered [13205.699636] Key type ._llcrypt registered [13205.708842] Key type .llcrypt registered [13205.871729] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13206.262336] Key type .llcrypt unregistered [13206.263728] Key type ._llcrypt unregistered [13268.677870] Key type ._llcrypt registered [13268.680116] Key type .llcrypt registered [13268.862448] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13269.437887] Key type .llcrypt unregistered [13269.447340] Key type ._llcrypt unregistered [13304.792731] Key type ._llcrypt registered [13304.794062] Key type .llcrypt registered [13304.945348] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13305.365432] Key type .llcrypt unregistered [13305.370163] Key type ._llcrypt unregistered [13341.127751] Key type ._llcrypt registered [13341.129656] Key type .llcrypt registered [13341.268937] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13341.594495] Key type .llcrypt unregistered [13341.596132] Key type ._llcrypt unregistered [13401.058145] Key type ._llcrypt registered [13401.059963] Key type .llcrypt registered [13401.171650] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [13417.210157] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [13418.081809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13418.204305] alg: No test for adler32 (adler32-zlib) [13419.280889] Lustre: Lustre: Build Version: 2.17.57_1_g2c0d074 [13419.538621] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [13421.247637] Key type lgssc registered [13422.759705] Lustre: Echo OBD driver; http://www.lustre.org/ [13458.990613] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13459.038807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13460.366400] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13460.397683] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13460.504228] Lustre: tst250fs-MDT0000: new disk, initializing [13460.643663] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13460.681490] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13466.104263] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13476.520484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13476.681274] Lustre: 374026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13476.795163] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13476.799185] Lustre: Skipped 1 previous similar message [13476.875687] Lustre: tst250fs-MDT0001: new disk, initializing [13476.978166] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13477.029066] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13477.049384] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13481.654160] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13486.677897] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13493.503696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13493.706452] Lustre: tst250fs-OST0000: new disk, initializing [13493.709604] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13493.714640] Lustre: 375652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13493.793443] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13494.465858] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13494.477871] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13494.616646] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13500.207707] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13510.448595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13510.584671] Lustre: tst250fs-OST0001: new disk, initializing [13510.587917] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13510.592914] Lustre: 376526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13510.684746] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13516.181866] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13519.430406] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13519.439856] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13519.524529] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13521.246579] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13523.897666] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13523.901415] Lustre: Skipped 2 previous similar messages [13543.393170] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13543.402988] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13543.420930] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13545.461884] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13545.465967] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13547.785864] Lustre: server umount tst250fs-MDT0000 complete [13550.520309] LustreError: 374017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786826675 with bad export cookie 17807043692275701054 [13550.522745] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13550.527179] LustreError: 374017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13550.564687] Lustre: tst250fs-MDT0001-lwp-OST0000: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13550.567088] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (stopping) [13550.573672] Lustre: Skipped 3 previous similar messages [13550.594971] Lustre: Skipped 4 previous similar messages [13550.814524] Lustre: server umount tst250fs-MDT0001 complete [13557.377980] Lustre: server umount tst250fs-OST0000 complete [13562.116280] Lustre: server umount tst250fs-OST0001 complete [13584.948369] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [13594.209741] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [13630.900175] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:45:54 (1786826754) [13632.567763] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13651.881100] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [13666.412658] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [13705.503599] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13404 sec ========= 16:47:08 (1786826828) [13708.513456] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:47:11 (1786826831) === [13713.777731] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:47:17 (1786826837) === [13762.841211] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13767.252170] Key type lgssc unregistered [13767.879039] LNet: 385680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13767.909029] LNetError: 385680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13768.938361] LNet: Removed LNI 192.168.201.118@tcp [13770.060832] Key type .llcrypt unregistered [13770.062568] Key type ._llcrypt unregistered