[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496375067 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002365] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007341] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010127] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.014054] *** VALIDATE selinux *** [ 0.022175] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026692] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027204] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028172] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029164] *** VALIDATE tmpfs *** [ 0.030616] *** VALIDATE proc *** [ 0.031357] *** VALIDATE cgroup *** [ 0.032022] *** VALIDATE cgroup2 *** [ 0.034212] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035186] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038021] Spectre V2 : User space: Vulnerable [ 0.039013] Speculative Store Bypass: Vulnerable [ 0.042699] debug: unmapping init [mem 0xffffffffb1059000-0xffffffffb1060fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045881] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046045] ... version: 2 [ 0.047024] ... bit width: 48 [ 0.048020] ... generic registers: 4 [ 0.049025] ... value mask: 0000ffffffffffff [ 0.050027] ... max period: 00007fffffffffff [ 0.051023] ... fixed-purpose events: 3 [ 0.052016] ... event mask: 000000070000000f [ 0.054302] rcu: Hierarchical SRCU implementation. [ 0.056838] smp: Bringing up secondary CPUs ... [ 0.057693] x86: Booting SMP configuration: [ 0.058040] .... node #0, CPUs: #1 #2 #3 [ 0.061547] smp: Brought up 1 node, 4 CPUs [ 0.063020] smpboot: Max logical packages: 1 [ 0.064029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.215037] node 0 deferred pages initialised in 149ms [ 0.217221] devtmpfs: initialized [ 0.219323] x86/mm: Memory block size: 128MB [ 0.222687] gcov: version magic: 0x41383552 [ 0.225340] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.229114] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.231326] pinctrl core: initialized pinctrl subsystem [ 0.233184] [ 0.233739] ************************************************************* [ 0.235024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.237017] ** ** [ 0.240016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.242015] ** ** [ 0.244017] ** This means that this kernel is built to expose internal ** [ 0.246016] ** IOMMU data structures, which may compromise security on ** [ 0.248020] ** your system. ** [ 0.250016] ** ** [ 0.252016] ** If you see this message and you are not debugging the ** [ 0.254017] ** kernel, report this immediately to your vendor! ** [ 0.256015] ** ** [ 0.258015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.260016] ************************************************************* [ 0.262763] NET: Registered protocol family 16 [ 0.265540] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.267072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.270081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.274013] cpuidle: using governor menu [ 0.275531] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.279166] PCI: Using configuration type 1 for base access [ 0.281177] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.291070] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.293024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.297187] cryptd: max_cpu_qlen set to 1000 [ 0.300217] ACPI: Added _OSI(Module Device) [ 0.302013] ACPI: Added _OSI(Processor Device) [ 0.303011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.305013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.309587] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.316157] ACPI: Interpreter enabled [ 0.317056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.319014] ACPI: Using IOAPIC for interrupt routing [ 0.322087] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.327390] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.338171] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.340039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.343017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.347083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.353520] acpiphp: Slot [2] registered [ 0.355221] acpiphp: Slot [5] registered [ 0.356107] acpiphp: Slot [6] registered [ 0.357106] acpiphp: Slot [7] registered [ 0.358136] acpiphp: Slot [8] registered [ 0.360139] acpiphp: Slot [9] registered [ 0.362130] acpiphp: Slot [10] registered [ 0.363230] acpiphp: Slot [3] registered [ 0.365192] acpiphp: Slot [4] registered [ 0.366420] acpiphp: Slot [11] registered [ 0.367245] acpiphp: Slot [12] registered [ 0.369148] acpiphp: Slot [13] registered [ 0.371184] acpiphp: Slot [14] registered [ 0.372150] acpiphp: Slot [15] registered [ 0.374145] acpiphp: Slot [16] registered [ 0.375240] acpiphp: Slot [17] registered [ 0.376131] acpiphp: Slot [18] registered [ 0.377131] acpiphp: Slot [19] registered [ 0.379122] acpiphp: Slot [20] registered [ 0.380152] acpiphp: Slot [21] registered [ 0.382201] acpiphp: Slot [22] registered [ 0.383228] acpiphp: Slot [23] registered [ 0.385128] acpiphp: Slot [24] registered [ 0.386216] acpiphp: Slot [25] registered [ 0.388150] acpiphp: Slot [26] registered [ 0.390136] acpiphp: Slot [27] registered [ 0.391161] acpiphp: Slot [28] registered [ 0.393134] acpiphp: Slot [29] registered [ 0.394137] acpiphp: Slot [30] registered [ 0.396149] acpiphp: Slot [31] registered [ 0.397105] PCI host bridge to bus 0000:00 [ 0.399035] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.401030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.403030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.405035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.408032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.410033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.412214] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.416544] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.420437] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.433015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.437669] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.441019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.444026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.448018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.452671] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.454824] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.457053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.460009] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.466020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.478018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.483022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.489655] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.498016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.502062] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.519018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.529334] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.546021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.554023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.575019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.589759] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.599018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.606017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.628015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.639154] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.657020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.675024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.724034] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.736673] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.749019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.757021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.771022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.778417] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.783027] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.789018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.804019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.815000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.817433] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.820431] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.823501] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.826297] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.830149] iommu: Default domain type: Passthrough [ 0.832404] SCSI subsystem initialized [ 0.834110] ACPI: bus type USB registered [ 0.835130] usbcore: registered new interface driver usbfs [ 0.837093] usbcore: registered new interface driver hub [ 0.839203] usbcore: registered new device driver usb [ 0.841182] pps_core: LinuxPPS API ver. 1 registered [ 0.843013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.845060] PTP clock support registered [ 0.847073] EDAC MC: Ver: 3.0.0 [ 0.848451] PCI: Using ACPI for IRQ routing [ 0.850769] NetLabel: Initializing [ 0.851011] NetLabel: domain hash size = 128 [ 0.853013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.855046] NetLabel: unlabeled traffic allowed by default [ 0.857187] vgaarb: loaded [ 0.859484] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.861019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.869436] clocksource: Switched to clocksource kvm-clock [ 0.986677] VFS: Disk quotas dquot_6.6.0 [ 0.989503] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.992584] *** VALIDATE ramfs *** [ 0.994058] *** VALIDATE hugetlbfs *** [ 0.995950] pnp: PnP ACPI init [ 0.998640] pnp: PnP ACPI: found 6 devices [ 1.017234] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.020969] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.024210] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.026759] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.029879] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.033163] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.037353] NET: Registered protocol family 2 [ 1.040207] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.045683] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.049631] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.055569] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.059671] TCP: Hash tables configured (established 65536 bind 65536) [ 1.062137] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.065433] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.068410] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.072273] NET: Registered protocol family 1 [ 1.075628] RPC: Registered named UNIX socket transport module. [ 1.078137] RPC: Registered udp transport module. [ 1.079828] RPC: Registered tcp transport module. [ 1.081583] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.083748] NET: Registered protocol family 44 [ 1.085646] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.088657] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.091321] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.094381] PCI: CLS 0 bytes, default 64 [ 1.096479] Unpacking initramfs... [ 2.557082] debug: unmapping init [mem 0xffff90953cc54000-0xffff90953ffbffff] [ 2.564134] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.566452] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.570787] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.114997] Initialise system trusted keyrings [ 3.116832] Key type blacklist registered [ 3.118946] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.130612] zbud: loaded [ 3.134134] *** VALIDATE nfs *** [ 3.135368] *** VALIDATE nfs4 *** [ 3.136863] pstore: using deflate compression [ 3.140228] Platform Keyring initialized [ 3.255706] NET: Registered protocol family 38 [ 3.257686] Key type asymmetric registered [ 3.259193] Asymmetric key parser 'x509' registered [ 3.261080] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.264151] io scheduler mq-deadline registered [ 3.266232] io scheduler kyber registered [ 3.268269] io scheduler bfq registered [ 3.270171] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.273486] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.276552] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.279526] ACPI: Power Button [PWRF] [ 3.286354] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.293639] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.306207] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.312676] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.331411] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.357844] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.389016] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.394173] Non-volatile memory driver v1.3 [ 3.395901] Linux agpgart interface v0.103 [ 3.429176] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.432798] vda: detected capacity change from 0 to 76689408 [ 3.448588] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.451876] vdb: detected capacity change from 0 to 1073741824 [ 3.467513] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.472488] vdc: detected capacity change from 0 to 2621440000 [ 3.489049] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.492172] vdd: detected capacity change from 0 to 2621440000 [ 3.507223] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.510488] vde: detected capacity change from 0 to 4294967296 [ 3.528572] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.531390] vdf: detected capacity change from 0 to 4294967296 [ 3.537756] libphy: Fixed MDIO Bus: probed [ 3.544138] usbcore: registered new interface driver usbserial_generic [ 3.546044] usbserial: USB Serial support registered for generic [ 3.547868] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.551413] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.552871] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.554892] mousedev: PS/2 mouse device common for all mice [ 3.558621] rtc_cmos 00:05: RTC can wake from S4 [ 3.558653] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.562928] rtc_cmos 00:05: registered as rtc0 [ 3.565652] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.570157] intel_pstate: CPU model not supported [ 3.571715] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.576267] hid: raw HID events driver (C) Jiri Kosina [ 3.577444] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.578520] usbcore: registered new interface driver usbhid [ 3.583121] usbhid: USB HID core driver [ 3.584792] drop_monitor: Initializing network drop monitor service [ 3.587560] Initializing XFRM netlink socket [ 3.589961] NET: Registered protocol family 10 [ 3.593095] Segment Routing with IPv6 [ 3.594723] NET: Registered protocol family 17 [ 3.597443] mpls_gso: MPLS GSO support [ 3.604234] RAS: Correctable Errors collector initialized. [ 3.606546] AVX version of gcm_enc/dec engaged. [ 3.608435] AES CTR mode by8 optimization enabled [ 3.695961] sched_clock: Marking stable (3695930599, 0)->(4583502966, -887572367) [ 3.699819] registered taskstats version 1 [ 3.701971] Loading compiled-in X.509 certificates [ 3.713547] zswap: loaded using pool lzo/zbud [ 3.759186] Key type big_key registered [ 3.772764] Key type encrypted registered [ 3.774804] ima: No TPM chip found, activating TPM-bypass! [ 3.777389] ima: Allocated hash algorithm: sha1 [ 3.779655] ima: No architecture policies found [ 3.781604] evm: Initialising EVM extended attributes: [ 3.783594] evm: security.selinux [ 3.784841] evm: security.ima [ 3.785875] evm: security.capability [ 3.786855] evm: HMAC attrs: 0x1 [ 3.789225] rtc_cmos 00:05: setting system clock to 2026-09-06 03:30:22 UTC (1788665422) [ 3.798464] debug: unmapping init [mem 0xffffffffb2003000-0xffffffffb21fffff] [ 3.803329] debug: unmapping init [mem 0xffffffffb0d82000-0xffffffffb1058fff] [ 3.813081] Write protecting the kernel read-only data: 28672k [ 3.816904] debug: unmapping init [mem 0xffffffffaf403000-0xffffffffaf5fffff] [ 3.820678] debug: unmapping init [mem 0xffffffffafd14000-0xffffffffafdfffff] [ 3.862516] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.872810] systemd[1]: Detected virtualization kvm. [ 3.875395] systemd[1]: Detected architecture x86-64. [ 3.877855] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.905945] systemd[1]: No hostname configured. [ 3.908218] systemd[1]: Set hostname to . [ 3.910763] random: systemd: uninitialized urandom read (16 bytes read) [ 3.913887] systemd[1]: Initializing machine ID from random generator. [ 4.057986] random: systemd: uninitialized urandom read (16 bytes read) [ 4.061937] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.066409] random: systemd: uninitialized urandom read (16 bytes read) [ 4.069369] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.074290] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.680718] device-mapper: uevent: version 1.0.3 [ 4.683265] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.406760] virtio_net virtio0 ens2: renamed from eth0 [ 5.412134] random: fast init done [ 5.450441] scsi host0: ata_piix [ 5.469481] scsi host1: ata_piix [ 5.471343] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.474054] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.078389] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.314588] random: crng init done [ 10.317436] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.608313] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.760972] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.023107] SELinux: Disabled at runtime. [ 12.083877] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.093529] systemd[1]: Detected virtualization kvm. [ 12.095759] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.597898] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.601495] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.606325] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.612301] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.615618] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.622424] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.626210] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ 12.793272] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 13.025446] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.361852] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.451708] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.579074] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.601741] EDAC sbridge: Ver: 1.1.2 [ 15.437986] Key type dns_resolver registered [ 15.765822] NFS: Registering the id_resolver key type [ 15.767783] Key type id_resolver registered [ 15.769454] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg630-server login: [ 42.160714] libcfs: loading out-of-tree module taints kernel. [ 42.182884] Key type ._llcrypt registered [ 42.184491] Key type .llcrypt registered [ 42.236361] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 52.722942] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 54.660174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.678380] alg: No test for adler32 (adler32-zlib) [ 56.139502] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 56.998725] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 58.767260] Key type lgssc registered [ 61.166298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.706974] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 121.571325] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 134.030289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 134.053594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.440112] hrtimer: interrupt took 2539419 ns [ 135.323299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 135.347565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 135.425848] Lustre: lustre-MDT0000: new disk, initializing [ 135.480082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 135.504902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 139.795856] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.740676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.842876] Lustre: 6477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 153.898959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 153.904593] Lustre: Skipped 1 previous similar message [ 154.034702] Lustre: lustre-MDT0001: new disk, initializing [ 154.101274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 154.131024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 154.147363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.659302] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.578753] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 175.246610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 175.574879] Lustre: lustre-OST0000: new disk, initializing [ 175.578931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 175.588584] Lustre: 8413:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 175.703376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.772652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 180.780207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 180.870462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 181.392262] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.232410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.366156] Lustre: lustre-OST0001: new disk, initializing [ 194.369305] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 194.375681] Lustre: 9484:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.448308] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.878807] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.287033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.299634] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 203.367299] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 209.738886] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 214.750785] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 220.507690] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing check_logdir /tmp/testlogs/ [ 224.897827] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing yml_node [ 229.527428] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 231.952786] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 234.102275] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 235.635507] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 23:34:12 EDT 2026 [ 250.155831] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 251.454354] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 266.722395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 266.734213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 266.750621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.807481] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.810886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.822307] Lustre: Skipped 2 previous similar messages [ 269.844386] Lustre: Skipped 3 previous similar messages [ 272.421900] Lustre: server umount lustre-MDT0000 complete [ 275.637035] LustreError: 7423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665694 with bad export cookie 4519087237935878131 [ 275.662604] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 275.663716] LustreError: 7423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 280.032749] LustreError: 12017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.032996] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.033598] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 280.051651] LustreError: 12017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 280.074963] Lustre: Skipped 1 previous similar message [ 282.241907] Lustre: server umount lustre-MDT0001 complete [ 286.243568] Lustre: server umount lustre-OST0000 complete [ 290.184874] Lustre: server umount lustre-OST0001 complete [ 296.346154] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 305.802563] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 344.931524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.267887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 345.315219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 345.397388] Lustre: lustre-MDT0000: new disk, initializing [ 345.516331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 345.547510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 350.363640] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 361.567513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 361.727056] Lustre: 15037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 361.823055] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 361.832706] Lustre: Skipped 1 previous similar message [ 361.953337] Lustre: lustre-MDT0001: new disk, initializing [ 362.060269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 362.118095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 362.135620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 366.423884] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 376.147692] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 381.969640] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 388.771951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 388.980699] Lustre: lustre-OST0000: new disk, initializing [ 388.987181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 388.999587] Lustre: 17008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 389.132187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.963632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 390.981799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 391.192877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 396.353739] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 406.026429] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 410.075168] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 410.269774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 414.984384] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 415.275196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 421.856888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 421.863083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 421.878803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 421.878830] Lustre: Skipped 1 previous similar message [ 421.916948] Lustre: Skipped 1 previous similar message [ 423.354532] Lustre: server umount lustre-OST0000 complete [ 426.976810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 427.005288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 427.012039] Lustre: Skipped 2 previous similar messages [ 432.789489] Lustre: server umount lustre-MDT0000 complete [ 436.234992] LustreError: 15028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665854 with bad export cookie 4519087237935879664 [ 436.235926] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 436.244242] LustreError: 15028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 436.799815] Lustre: server umount lustre-MDT0001 complete [ 442.679335] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:37:39 (1788665859) [ 459.896063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.393648] LustreError: 19882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 460.462732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.606902] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 465.894332] LustreError: 19883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.029672] LustreError: 19882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.005985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.417678] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 486.332915] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 491.458673] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 493.217304] Lustre: Failing over lustre-MDT0000 [ 493.479626] Lustre: server umount lustre-MDT0000 complete [ 496.096809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 496.096975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.099080] LustreError: 19878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.125265] Lustre: Skipped 2 previous similar messages [ 505.319431] LustreError: 20610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.346349] LustreError: 20610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 510.867862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.987729] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.451290] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 516.581260] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 516.584724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 516.601442] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 523.899771] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 530.914946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.929531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.934019] Lustre: Skipped 3 previous similar messages [ 533.804599] Lustre: server umount lustre-MDT0000 complete [ 536.040694] LustreError: 19882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.058050] LustreError: 19882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 537.516591] LustreError: 19864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665956 with bad export cookie 4519087237935881078 [ 537.527930] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.532507] LustreError: 19864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 537.857930] Lustre: server umount lustre-MDT0001 complete [ 547.664749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.277316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 548.282820] Lustre: Skipped 2 previous similar messages [ 552.553363] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 561.648717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.079656] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 573.585951] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 579.197440] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 581.639307] Lustre: Failing over lustre-MDT0000 [ 582.074582] Lustre: server umount lustre-MDT0000 complete [ 582.625590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 582.627553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 582.646623] LustreError: 23216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.656073] Lustre: Skipped 2 previous similar messages [ 582.670653] LustreError: 23216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 599.010702] Lustre: 3610:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788666001/real 1788666001] req@ffff9095bf816d80 x1875551689292544/t0(0) o400->MGC192.168.206.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788666017 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 599.040196] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 600.610851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.251432] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3eb705e6562b5f20 [ 609.268121] Lustre: MGC192.168.206.130@tcp: Connection restored to 0@lo (at 0@lo) [ 609.277924] Lustre: Skipped 1 previous similar message [ 609.562497] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 613.616860] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 614.883161] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 614.891315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 614.917464] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 622.155192] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 630.243926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 630.248235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 630.257042] Lustre: Skipped 1 previous similar message [ 632.010471] Lustre: server umount lustre-MDT0000 complete [ 635.179785] LustreError: 23197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666053 with bad export cookie 4519087237935882016 [ 635.182461] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.189132] LustreError: 23197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 635.460847] Lustre: server umount lustre-MDT0001 complete [ 642.934861] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:41:00 (1788666060) [ 658.627630] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 668.617512] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 708.042332] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 718.736291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 719.029646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 719.058405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 719.140837] Lustre: lustre-MDT0000: new disk, initializing [ 719.217825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.224791] Lustre: Skipped 2 previous similar messages [ 719.246913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 723.973413] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 735.052736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.134284] Lustre: 29594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 735.395991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 739.252402] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 749.467174] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 755.201443] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 762.689303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 763.157323] Lustre: 31562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 765.223869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 765.288752] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 765.385726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 765.410564] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 772.269963] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 782.663917] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 788.775405] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 811.492721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 811.494709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 811.511154] Lustre: Skipped 2 previous similar messages [ 811.516562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 811.529799] Lustre: Skipped 1 previous similar message [ 814.646554] Lustre: server umount lustre-OST0000 complete [ 824.805737] Lustre: server umount lustre-MDT0000 complete [ 826.850053] LustreError: 29605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 826.866359] LustreError: 29605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 828.327871] LustreError: 29586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666247 with bad export cookie 4519087237935882429 [ 828.330256] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 828.340141] LustreError: 29586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 828.669330] Lustre: server umount lustre-MDT0001 complete [ 839.738508] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 843.112900] Key type lgssc unregistered [ 843.459647] LNet: 33536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.476150] LNetError: 33536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.487621] LNet: Removed LNI 192.168.206.130@tcp [ 844.743259] Key type .llcrypt unregistered [ 844.749612] Key type ._llcrypt unregistered [ 869.700580] Key type ._llcrypt registered [ 869.702945] Key type .llcrypt registered [ 869.788463] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 871.072622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 871.266490] alg: No test for adler32 (adler32-zlib) [ 872.408699] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 872.614036] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 874.311237] Key type lgssc registered [ 875.855339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 883.865770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 883.889719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.392566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.237974] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 895.970626] LustreError: 34946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 895.981328] LustreError: 34946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 899.624021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.087847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 905.254489] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 914.528218] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 920.498520] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 928.723332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 929.128752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.243755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 937.345689] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 945.550221] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 946.714468] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 962.603314] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:46:19 (1788666379) [ 965.293990] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 967.604571] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 974.914724] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:46:31 (1788666391) [ 983.601694] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:46:40 (1788666400) [ 984.732558] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 994.394530] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 994.405096] Lustre: Skipped 2 previous similar messages [ 996.559952] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:46:53 (1788666413) [ 998.893878] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 998.899848] Lustre: Skipped 2 previous similar messages [ 1007.180729] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1007.194262] Lustre: Skipped 4 previous similar messages [ 1022.908680] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:47:19 (1788666439) [ 1026.176370] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1029.575144] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1038.627326] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:47:35 (1788666455) [ 1040.756502] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1040.763668] Lustre: Skipped 6 previous similar messages [ 1056.809542] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:47:53 (1788666473) [ 1073.633768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1073.665085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1073.671336] Lustre: Skipped 1 previous similar message [ 1078.763610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1078.771351] Lustre: Skipped 1 previous similar message [ 1079.505840] Lustre: server umount lustre-OST0000 complete [ 1083.873495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.893350] Lustre: Skipped 1 previous similar message [ 1083.900423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1083.913320] Lustre: Skipped 1 previous similar message [ 1088.996017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.011612] Lustre: Skipped 1 previous similar message [ 1089.286967] Lustre: server umount lustre-MDT0000 complete [ 1094.114442] LustreError: 34941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.140851] LustreError: 34941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1094.576323] LustreError: 34927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666513 with bad export cookie 9978715108895121171 [ 1094.593115] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.594291] LustreError: 34927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1095.467948] Lustre: server umount lustre-MDT0001 complete [ 1111.185631] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 1114.980163] Key type lgssc unregistered [ 1115.289785] LNet: 42808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.307535] LNetError: 42808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.344284] LNet: Removed LNI 192.168.206.130@tcp [ 1116.577643] Key type .llcrypt unregistered [ 1116.579786] Key type ._llcrypt unregistered [ 1136.103966] Key type ._llcrypt registered [ 1136.107116] Key type .llcrypt registered [ 1136.190815] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 1149.599903] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 1150.915758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1150.965935] alg: No test for adler32 (adler32-zlib) [ 1151.978730] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1152.253329] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 1153.943927] Key type lgssc registered [ 1155.128524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1194.225655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1194.235977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.424263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1195.462858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1195.584749] Lustre: lustre-MDT0000: new disk, initializing [ 1195.687035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.705425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1199.875328] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1210.900824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.079961] Lustre: 45960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1211.156789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1211.161239] Lustre: Skipped 1 previous similar message [ 1211.258081] Lustre: lustre-MDT0001: new disk, initializing [ 1211.389320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1211.417350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1211.429723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1215.328635] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1223.153319] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1227.452707] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1233.284432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1233.472687] Lustre: lustre-OST0000: new disk, initializing [ 1233.476368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1233.483229] Lustre: 47929:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1233.546918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.388365] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1242.696254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1242.703606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1242.803267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1246.353341] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1251.513826] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1251.743746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1257.227646] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1257.453755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1263.584731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1263.591174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.593828] LustreError: Skipped 1 previous similar message [ 1263.596134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1263.629758] Lustre: Skipped 1 previous similar message [ 1265.684894] Lustre: server umount lustre-OST0000 complete [ 1272.802585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1272.808061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1272.820485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.830140] Lustre: Skipped 1 previous similar message [ 1273.827463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.828636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1273.856028] Lustre: Skipped 1 previous similar message [ 1275.296343] Lustre: server umount lustre-MDT0000 complete [ 1278.586404] LustreError: 45950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666697 with bad export cookie 15264799030938030331 [ 1278.587268] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1278.605549] LustreError: 45950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1278.917713] Lustre: server umount lustre-MDT0001 complete [ 1287.039431] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 23:51:44 (1788666704) [ 1293.047479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.591853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.175476] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1298.915603] LustreError: 50388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.945227] LustreError: 50388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1304.048977] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.114608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.387756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1310.752405] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1317.605220] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1323.493770] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1329.775558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1330.107320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.297719] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1343.774541] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1348.457377] LustreError: 52917:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1349.368285] Lustre: Modifying parameter general.timeout=1 in log params [ 1351.218483] Lustre: Modifying parameter general.timeout=3 in log params [ 1351.222196] Lustre: Skipped 1 previous similar message [ 1353.694564] Lustre: Modifying parameter general.timeout=6 in log params [ 1353.699716] Lustre: Skipped 2 previous similar messages [ 1357.741647] Lustre: Modifying parameter general.timeout=11 in log params [ 1357.746690] Lustre: Skipped 4 previous similar messages [ 1365.886410] Lustre: Modifying parameter general.timeout=22 in log params [ 1365.892663] Lustre: Skipped 10 previous similar messages [ 1382.449906] Lustre: Modifying parameter general.timeout=44 in log params [ 1382.456687] Lustre: Skipped 21 previous similar messages [ 1414.542364] Lustre: Modifying parameter general.timeout=84 in log params [ 1414.547420] Lustre: Skipped 39 previous similar messages [ 1463.471785] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 23:54:40 (1788666880) [ 1465.815327] Lustre: 59846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1467.637633] Lustre: 59957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1469.645443] Lustre: 60056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1473.864102] Lustre: 60254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1473.881176] Lustre: 60254:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1480.372271] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1480.384618] Lustre: Skipped 54 previous similar messages [ 1501.989191] Lustre: 61726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1501.995685] Lustre: 61726:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1502.021813] LustreError: 61726:0:(mgs_handler.c:1205:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1506.309294] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 23:55:23 (1788666923) [ 1521.633095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1521.637319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.647843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.838916] Lustre: server umount lustre-MDT0000 complete [ 1527.267192] LustreError: 50383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.290924] LustreError: 50383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1527.568547] LustreError: 51123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666946 with bad export cookie 15264799030938031220 [ 1527.573655] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1527.577506] LustreError: 51123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1527.926512] Lustre: server umount lustre-MDT0001 complete [ 1542.030202] Lustre: server umount lustre-OST0000 complete [ 1563.808191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.107941] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1564.392073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.315977] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1575.897127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.976247] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1576.018488] Lustre: 64259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1580.153267] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1586.513396] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1591.060104] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1597.372700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1597.694149] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1598.116137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.125726] Lustre: Skipped 1 previous similar message [ 1603.981732] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1611.685972] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1613.345630] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1613.353379] Lustre: Skipped 14 previous similar messages [ 1613.404831] Lustre: 66823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1619.944835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1619.953914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.972279] Lustre: Skipped 2 previous similar messages [ 1619.981174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1619.992295] Lustre: Skipped 3 previous similar messages [ 1630.691263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1630.697088] Lustre: Skipped 4 previous similar messages [ 1634.271720] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1634.444822] Lustre: server umount lustre-OST0000 complete [ 1637.857151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1637.866891] LustreError: Skipped 1 previous similar message [ 1637.873197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.885763] Lustre: Skipped 1 previous similar message [ 1644.006828] Lustre: server umount lustre-MDT0000 complete [ 1646.048752] LustreError: 64270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.059697] LustreError: 64270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1647.472093] LustreError: 66119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667066 with bad export cookie 15264799030938034139 [ 1647.478384] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1647.491979] LustreError: 66119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1647.791264] Lustre: server umount lustre-MDT0001 complete [ 1658.189965] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 1660.909604] Key type lgssc unregistered [ 1661.234663] LNet: 68031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.242889] LNetError: 68031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.273045] LNet: Removed LNI 192.168.206.130@tcp [ 1662.244316] Key type .llcrypt unregistered [ 1662.248512] Key type ._llcrypt unregistered [ 1683.606944] Key type ._llcrypt registered [ 1683.609893] Key type .llcrypt registered [ 1683.709666] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 1684.491243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1684.525514] alg: No test for adler32 (adler32-zlib) [ 1685.672812] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1685.903611] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 1687.551157] Key type lgssc registered [ 1688.661216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1695.596198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1695.635747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.205367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1701.350627] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1706.471841] LustreError: 69443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.485550] LustreError: 69443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1709.417206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1710.173322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1715.134259] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1722.271425] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1727.601873] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1734.237881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1734.782963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1740.149956] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1748.058641] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1756.769439] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1766.371536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1766.378910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.397217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.910453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.911118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.947199] Lustre: Skipped 1 previous similar message [ 1767.956601] Lustre: Skipped 2 previous similar messages [ 1770.653831] Lustre: server umount lustre-MDT0000 complete [ 1773.033839] LustreError: 70181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.069061] LustreError: 70181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1774.010518] LustreError: 69422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667192 with bad export cookie 2087056986587271020 [ 1774.022170] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1774.029095] LustreError: 69422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1774.286402] Lustre: server umount lustre-MDT0001 complete [ 1788.425543] Lustre: server umount lustre-OST0000 complete [ 1815.129695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.515444] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1815.574509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1815.893477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1820.242526] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1828.619896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.711627] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1828.751757] Lustre: 74313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1829.254401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1833.701370] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1841.161528] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1847.268440] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1854.437698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1854.726584] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1855.163420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.217157] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1868.722899] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1870.367054] Lustre: 76902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1872.130914] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1878.501661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1878.507886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1878.539117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.258599] Lustre: server umount lustre-OST0000 complete [ 1889.249700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1889.258183] Lustre: Skipped 1 previous similar message [ 1889.260956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1889.263564] Lustre: Skipped 2 previous similar messages [ 1892.850770] Lustre: server umount lustre-MDT0000 complete [ 1894.369879] LustreError: 74325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.394105] LustreError: 74325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1896.335587] LustreError: 74306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667315 with bad export cookie 2087056986587271860 [ 1896.342394] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.350816] LustreError: 74306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1896.632692] Lustre: server umount lustre-MDT0001 complete [ 1907.222116] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 1909.643661] Key type lgssc unregistered [ 1909.972723] LNet: 78112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.981140] LNetError: 78112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.000701] LNet: Removed LNI 192.168.206.130@tcp [ 1911.027941] Key type .llcrypt unregistered [ 1911.036160] Key type ._llcrypt unregistered [ 1932.270415] Key type ._llcrypt registered [ 1932.274820] Key type .llcrypt registered [ 1932.369766] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 1933.122416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1933.141417] alg: No test for adler32 (adler32-zlib) [ 1934.226322] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1934.533585] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 1936.273560] Key type lgssc registered [ 1937.326743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1944.135789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1944.166255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.718665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.941722] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1955.306190] LustreError: 79502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.327328] LustreError: 79502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1957.291030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.737853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1961.561050] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1967.096863] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1971.663541] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1977.927503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1978.361191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.025369] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 1991.813723] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1997.885410] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1998.598701] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2000.155474] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2000.162233] Lustre: Skipped 1 previous similar message [ 2002.706282] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2002.714261] Lustre: Skipped 2 previous similar messages [ 2007.132858] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2007.141222] Lustre: Skipped 4 previous similar messages [ 2015.854317] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2015.873303] Lustre: Skipped 9 previous similar messages [ 2032.130674] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2032.139101] Lustre: Skipped 19 previous similar messages [ 2064.421566] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2064.428056] Lustre: Skipped 37 previous similar messages [ 2091.840817] Lustre: 87596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2094.863460] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:05:11 (1788667511) [ 2097.070952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2099.342149] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:05:15 (1788667515) [ 2105.317494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2105.342327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.370191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2108.387270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2108.395180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2108.401214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.421186] Lustre: Skipped 1 previous similar message [ 2109.950354] Lustre: server umount lustre-OST0000 complete [ 2116.576972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2116.585297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.605993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2118.625189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2118.631258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.745374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.753042] Lustre: Skipped 2 previous similar messages [ 2128.351137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2128.668761] Lustre: server umount lustre-MDT0000 complete [ 2128.865622] LustreError: 79496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.889135] LustreError: 79496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2133.115429] LustreError: 81351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667551 with bad export cookie 8225485793458072607 [ 2133.118354] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2133.127412] LustreError: 81351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2133.520085] Lustre: server umount lustre-MDT0001 complete [ 2142.578590] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 2144.796451] Key type lgssc unregistered [ 2145.030585] LNet: 88846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2145.044213] LNetError: 88846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2145.062978] LNet: Removed LNI 192.168.206.130@tcp [ 2145.795879] Key type .llcrypt unregistered [ 2145.799731] Key type ._llcrypt unregistered [ 2149.501434] Key type ._llcrypt registered [ 2149.507944] Key type .llcrypt registered [ 2149.659324] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2153.572905] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules [ 2154.324677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2154.419299] alg: No test for adler32 (adler32-zlib) [ 2155.421748] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 2155.673832] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 2157.351152] Key type lgssc registered [ 2158.508903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2159.519728] LustreError: 90043:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2176.179442] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 2181.325814] LustreError: 90043:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2184.583758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2184.597836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.133898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.328328] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2195.425863] LustreError: 91068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.448590] LustreError: 91068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2200.547722] LustreError: 91069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.785753] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:06:57 (1788667617) [ 2205.669094] LustreError: 91068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.714648] LustreError: 91100:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2205.724911] LustreError: 91100:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2205.730807] LustreError: 91100:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2206.023151] Lustre: server umount lustre-MDT0000 complete [ 2218.805764] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 2221.235933] Key type lgssc unregistered [ 2221.546110] LNet: 92594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.560071] LNetError: 92594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.575505] LNet: Removed LNI 192.168.206.130@tcp [ 2222.327320] Key type .llcrypt unregistered [ 2222.330066] Key type ._llcrypt unregistered [ 2239.384867] Key type ._llcrypt registered [ 2239.388626] Key type .llcrypt registered [ 2239.495648] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 2240.600415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2240.651095] alg: No test for adler32 (adler32-zlib) [ 2241.741503] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 2242.024690] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 2243.697485] Key type lgssc registered [ 2244.663545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2252.470906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2252.509577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.103393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.002363] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2263.531752] LustreError: 93809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.574379] LustreError: 93809:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2268.338711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.641267] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2269.236391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2273.517577] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2280.201371] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2285.448760] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2291.118531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2291.708066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2296.797641] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2302.493963] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2344.192391] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:09:21 (1788667761) [ 2345.780550] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2347.369397] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:09:24 (1788667764) [ 2355.174037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.175668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.184299] Lustre: Skipped 1 previous similar message [ 2355.192321] Lustre: Skipped 2 previous similar messages [ 2356.984589] Lustre: server umount lustre-MDT0000 complete [ 2359.828249] LustreError: 93788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667778 with bad export cookie 9971751026709128362 [ 2359.828530] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2359.839591] LustreError: 93788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2360.132205] Lustre: server umount lustre-MDT0001 complete [ 2373.517812] Lustre: server umount lustre-OST0000 complete [ 2384.932470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.433478] LustreError: 98081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.545412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.130962] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2391.019822] LustreError: 98082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.131696] LustreError: 98081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.377880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.449699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2405.633823] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2413.483231] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2418.855777] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2431.949222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2432.122885] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2432.128761] LustreError: 98070:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2432.148096] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2432.154735] LustreError: 100166:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -98 [ 2432.161710] LustreError: 100166:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 2432.166568] LustreError: 100166:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2432.283783] Lustre: server umount lustre-OST0000 complete [ 2432.285300] LustreError: 100166:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2439.776993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2439.903750] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2439.910820] LustreError: 98070:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2439.921901] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2439.931691] LustreError: 100626:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -98 [ 2439.938260] LustreError: 100626:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 2439.945688] LustreError: 100626:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2440.005063] Lustre: server umount lustre-OST0000 complete [ 2440.009528] LustreError: 100626:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2449.504882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2449.718128] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2449.725816] Lustre: Found index 0 for lustre-OST0000, updating log [ 2449.740945] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2449.817177] Lustre: lustre-OST0000: new disk, initializing [ 2449.820686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2449.829121] Lustre: 101161:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2450.094681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.563696] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2460.098088] Lustre: Failing over lustre-OST0000 [ 2460.297402] Lustre: server umount lustre-OST0000 complete [ 2466.278401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2466.288100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.308697] Lustre: Skipped 1 previous similar message [ 2466.318140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2469.858817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2469.871602] Lustre: Skipped 2 previous similar messages [ 2474.094119] Lustre: server umount lustre-MDT0000 complete [ 2474.976263] LustreError: 98076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.007987] LustreError: 98076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2477.826624] LustreError: 101174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667896 with bad export cookie 9971751026709131106 [ 2477.831812] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2477.835380] LustreError: 101174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2478.199051] Lustre: server umount lustre-MDT0001 complete [ 2488.072411] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:11:44 (1788667904) [ 2504.650424] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 2514.919538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.193248] LustreError: 103508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.346684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2518.844213] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2527.442790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.190963] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2534.816711] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2540.917362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2541.693540] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2547.780349] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2552.936793] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2552.945314] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2552.958279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2556.066082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2556.176766] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2556.245591] Lustre: lustre-OST0001: new disk, initializing [ 2556.249858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2556.255099] Lustre: 105827:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2556.497094] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 2556.519690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2556.534421] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2556.544042] Lustre: Skipped 2 previous similar messages [ 2559.585808] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2559.596765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2559.618683] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2559.619551] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2559.635206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2561.928915] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2563.189647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2563.201921] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2563.280534] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2569.125241] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2576.224071] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2584.042409] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.052736] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2584.078275] Lustre: Skipped 3 previous similar messages [ 2584.086627] Lustre: Skipped 1 previous similar message [ 2586.806532] Lustre: server umount lustre-MDT0001 complete [ 2589.165356] LustreError: 103508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.187788] LustreError: 103508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2596.659983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.750491] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2596.757299] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2596.805239] Lustre: 103498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2598.516943] LustreError: 107382:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2598.526843] LustreError: 107382:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2598.541250] LustreError: 103501:0:(mgc_request.c:612:do_requeue()) failed processing log: -17 [ 2598.683334] LustreError: 107402:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.130@tcp: cfg command failed: rc = -17 [ 2598.690772] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2598.690772] [ 2602.309590] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2603.000753] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2612.587736] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:13:49 (1788668029) [ 2632.407609] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 2650.286115] Lustre: 109441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2672.768496] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2676.877296] Lustre: 110577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2771.423920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2771.439122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.460858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2771.463476] Lustre: Skipped 2 previous similar messages [ 2775.156885] Lustre: server umount lustre-MDT0000 complete [ 2777.057619] LustreError: 103505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.069938] LustreError: 103505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2778.574315] LustreError: 103491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668197 with bad export cookie 9971751026709132037 [ 2778.577726] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2778.583179] LustreError: 103491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2778.850249] Lustre: server umount lustre-MDT0001 complete [ 2793.273437] Lustre: server umount lustre-OST0000 complete [ 2807.789392] Lustre: server umount lustre-OST0001 complete [ 2813.585669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2824.529409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2832.986602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2843.056897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2861.184370] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 2873.199566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.225260] Lustre: lustre-MDT0000: reset Object Index mappings [ 2873.681183] LustreError: 114535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.715916] LustreError: 114535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2873.832099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.841301] Lustre: Skipped 1 previous similar message [ 2878.556849] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2886.449792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.470732] Lustre: lustre-MDT0001: reset Object Index mappings [ 2891.630031] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2894.433529] Lustre: 115681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2900.658601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.546791] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2911.547939] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2916.856481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2918.556155] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2918.565689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 2918.574653] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2924.711761] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 2934.701134] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2939.687036] Lustre: 117563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2991.875691] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:20:08 (1788668408) [ 3020.257092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3020.264484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.278616] Lustre: Skipped 3 previous similar messages [ 3020.290235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.295748] Lustre: Skipped 3 previous similar messages [ 3024.827854] Lustre: server umount lustre-MDT0000 complete [ 3026.426759] LustreError: 115270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.458212] LustreError: 115270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3028.325724] LustreError: 114515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668447 with bad export cookie 9971751026709179133 [ 3028.334460] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3028.345326] LustreError: 114515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3031.532359] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.542670] Lustre: Skipped 4 previous similar messages [ 3034.890414] Lustre: server umount lustre-MDT0001 complete [ 3044.911393] Lustre: server umount lustre-OST0000 complete [ 3063.263563] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3063.449342] Lustre: server umount lustre-OST0001 complete [ 3070.239528] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 3080.963478] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 3117.000471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.237455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3117.290246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3117.390197] Lustre: lustre-MDT0000: new disk, initializing [ 3117.461746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.470726] Lustre: Skipped 3 previous similar messages [ 3117.482096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3122.123710] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3128.325277] Lustre: server umount lustre-MDT0000 complete [ 3136.487109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.864556] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3136.891781] Lustre: 121832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3136.902766] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3141.194494] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3145.890691] Lustre: server umount lustre-MDT0000 complete [ 3154.925769] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:22:51 (1788668571) [ 3156.879423] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3159.028553] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:22:55 (1788668575) [ 3176.829311] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 3185.851758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.573358] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3198.563481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.667185] Lustre: 123898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3198.722951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3198.727215] Lustre: Skipped 1 previous similar message [ 3198.804479] Lustre: lustre-MDT0001: new disk, initializing [ 3198.910068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3198.924512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3203.209495] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3207.970714] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3214.192757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3214.443408] Lustre: lustre-OST0000: new disk, initializing [ 3214.448367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3214.455542] Lustre: 125475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3216.511035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3216.524102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3216.664946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3220.627832] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3231.247429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3231.401276] Lustre: lustre-OST0001: new disk, initializing [ 3231.408922] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3231.414619] Lustre: 126343:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3233.264507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3233.274820] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3233.382853] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3238.315952] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 3247.949541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3251.392243] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3611.881891] Lustre: 132753:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3611.907340] Lustre: 132753:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3611.919834] Lustre: 132753:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3611.937919] Lustre: 132753:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3611.948559] Lustre: 132753:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3611.955655] Lustre: 132753:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3897.354562] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:35:13 (1788669313) [ 3914.214180] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.217978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3914.229048] Lustre: Skipped 2 previous similar messages [ 3914.233394] Lustre: Skipped 9 previous similar messages [ 3919.332507] LustreError: 127161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3919.343212] LustreError: 127161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3919.438641] Lustre: server umount lustre-MDT0000 complete [ 3922.439445] LustreError: 123889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669341 with bad export cookie 9971751026709247663 [ 3922.442395] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3922.454490] LustreError: 123889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3924.471559] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3924.498552] Lustre: Skipped 1 previous similar message [ 3924.507122] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3924.512241] Lustre: Skipped 2 previous similar messages [ 3929.073858] Lustre: server umount lustre-MDT0001 complete [ 3938.977681] Lustre: server umount lustre-OST0000 complete [ 3956.191295] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3956.366736] Lustre: server umount lustre-OST0001 complete [ 3962.195847] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 3970.918453] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 4002.792340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4003.010821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4003.035042] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4003.102154] Lustre: lustre-MDT0000: new disk, initializing [ 4003.165759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4003.171532] Lustre: Skipped 5 previous similar messages [ 4003.190123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4006.778231] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 4015.548429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4015.669327] Lustre: 135808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4015.749522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4015.756722] Lustre: Skipped 1 previous similar message [ 4015.863234] Lustre: lustre-MDT0001: new disk, initializing [ 4015.975580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4015.988906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4019.877450] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 4028.794964] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4034.093787] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4039.586789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4039.997347] Lustre: lustre-OST0000: new disk, initializing [ 4040.001673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4040.009927] Lustre: 137773:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4040.083263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4040.087510] Lustre: Skipped 1 previous similar message [ 4042.033102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4042.056327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4042.162142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4045.682473] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 4054.632839] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4064.275963] Lustre: lustre-MDD0000: changelog on [ 4066.830828] Lustre: lustre-MDD0001: changelog on [ 5904.305632] Lustre: lustre-MDD0001: changelog off [ 5906.656065] Lustre: 141657:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5907.848435] Lustre: lustre-MDD0000: changelog off [ 5915.616257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5915.628058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5915.641373] Lustre: Skipped 1 previous similar message [ 5915.658568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5915.671019] Lustre: Skipped 2 previous similar messages [ 5920.904364] Lustre: server umount lustre-OST0000 complete [ 5925.858308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5925.867435] LustreError: Skipped 1 previous similar message [ 5925.870948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.884349] Lustre: Skipped 1 previous similar message [ 5925.887822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5925.896195] Lustre: Skipped 2 previous similar messages [ 5931.212323] Lustre: server umount lustre-MDT0000 complete [ 5931.494396] LustreError: 139582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.506691] LustreError: 139582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5935.193312] LustreError: 135799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671353 with bad export cookie 9971751026710065746 [ 5935.206382] LustreError: 135799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5935.207344] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5935.543810] Lustre: server umount lustre-MDT0001 complete [ 5947.535850] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 5950.586235] Key type lgssc unregistered [ 5950.911870] LNet: 146923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5950.920536] LNetError: 146923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5950.948359] LNet: Removed LNI 192.168.206.130@tcp [ 5952.054248] Key type .llcrypt unregistered [ 5952.056277] Key type ._llcrypt unregistered [ 5976.633753] Key type ._llcrypt registered [ 5976.637115] Key type .llcrypt registered [ 5976.748081] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 5993.376391] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 5994.563848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5994.701087] alg: No test for adler32 (adler32-zlib) [ 5995.749781] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 5996.041115] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 5997.719225] Key type lgssc registered [ 5998.969161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6037.810504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6037.836049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6039.124631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6039.176817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6039.255619] Lustre: lustre-MDT0000: new disk, initializing [ 6039.342941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6039.366614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6043.796170] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6054.037610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6054.221403] Lustre: 150122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6054.284667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6054.289231] Lustre: Skipped 1 previous similar message [ 6054.365274] Lustre: lustre-MDT0001: new disk, initializing [ 6054.443375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6054.473468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6054.486952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6059.116837] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6068.583498] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6073.599473] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6080.472141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6080.672518] Lustre: lustre-OST0000: new disk, initializing [ 6080.677845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6080.684143] Lustre: 152090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6080.842480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6086.216607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6086.230067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6086.337358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6087.000924] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6095.928808] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6101.265524] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_module obdecho/obdecho [ 6103.345644] LustreError: 153092:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6103.354626] LustreError: 153092:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6105.087931] LustreError: 153188:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6117.348560] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6117.361014] Lustre: Skipped 1 previous similar message [ 6117.362954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6117.375627] Lustre: Skipped 2 previous similar messages [ 6122.473364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6122.486950] Lustre: Skipped 2 previous similar messages [ 6122.866861] Lustre: server umount lustre-MDT0000 complete [ 6126.704913] LustreError: 150915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671545 with bad export cookie 3525530983374151923 [ 6126.720016] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6127.040539] Lustre: server umount lustre-MDT0001 complete [ 6140.871473] Lustre: server umount lustre-OST0000 complete [ 6162.601870] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 6171.032679] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 6211.040787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.310661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6211.374392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6211.473531] Lustre: lustre-MDT0000: new disk, initializing [ 6211.580432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.606286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6216.099823] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6225.960836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6226.084536] Lustre: 156906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6226.153330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6226.158700] Lustre: Skipped 1 previous similar message [ 6226.278549] Lustre: lustre-MDT0001: new disk, initializing [ 6226.408788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6226.448277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6226.466491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6230.777676] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6240.016240] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6246.273590] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6252.938551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6253.188190] Lustre: lustre-OST0000: new disk, initializing [ 6253.193645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6253.209518] Lustre: 158875:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6253.316990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6258.445231] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6261.393171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6261.413061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6261.545039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6268.187204] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6274.079751] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6274.349954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6279.953376] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6280.212618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6282.209294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6282.210292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6282.245134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6282.267677] Lustre: Skipped 1 previous similar message [ 6287.331119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6287.335806] Lustre: Skipped 1 previous similar message [ 6288.227562] Lustre: server umount lustre-OST0000 complete [ 6292.455919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6292.487600] Lustre: Skipped 1 previous similar message [ 6292.506299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6292.516527] Lustre: Skipped 1 previous similar message [ 6298.483369] Lustre: server umount lustre-MDT0000 complete [ 6302.811725] LustreError: 156899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671721 with bad export cookie 3525530983374152882 [ 6302.816988] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6302.821206] LustreError: 156899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6303.250863] Lustre: server umount lustre-MDT0001 complete [ 6306.425795] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:15:23 (1788671723) [ 6307.666263] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6309.277268] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:15:26 (1788671726) [ 6316.537372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.936881] LustreError: 161285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.962714] LustreError: 161285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6317.066295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6321.284926] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6322.147781] LustreError: 161286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6322.164345] LustreError: 161286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6327.271770] LustreError: 161285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6329.450872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.734297] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6340.519410] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6346.026411] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6352.056250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6352.422928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6352.432384] Lustre: Skipped 1 previous similar message [ 6358.797489] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6365.878355] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6371.756500] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.30@tcp (stopping) [ 6371.774659] Lustre: Skipped 2 previous similar messages [ 6371.810384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.842849] Lustre: Skipped 3 previous similar messages [ 6377.217603] Lustre: server umount lustre-MDT0000 complete [ 6380.814100] LustreError: 161268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671799 with bad export cookie 3525530983374153764 [ 6380.819166] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6380.823844] LustreError: 161268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6381.962483] LustreError: 161280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6382.052938] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.081103] LustreError: 163589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6387.111067] LustreError: 163589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6390.241193] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6390.250208] Lustre: Skipped 11 previous similar messages [ 6395.359132] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6395.642758] Lustre: server umount lustre-MDT0001 complete [ 6401.624829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6412.326463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.488220] LustreError: 165125:0:(llog.c:1655:llog_backup()) MGC192.168.206.130@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6412.494751] Lustre: 165125:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.130@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6421.905607] LustreError: 165130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.918224] LustreError: 165130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6438.287964] LustreError: 165131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6438.298372] LustreError: 165131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 6470.627815] LustreError: 165130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6470.653266] LustreError: 165130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6473.184422] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6473.194560] Lustre: 165125:0:(mgc_request.c:1898:mgc_process_log()) MGC192.168.206.130@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6514.161745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6518.371478] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6519.280171] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6526.512272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.652861] LustreError: 165131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.657195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6535.661306] LustreError: 165131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 6552.044331] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x30ed33c014a6817d [ 6552.060814] Lustre: MGC192.168.206.130@tcp: Connection restored to 0@lo (at 0@lo) [ 6556.166321] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6557.678872] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6562.575989] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6564.231946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6569.059730] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6570.358815] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6588.390992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6588.391334] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.392978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6588.392984] Lustre: Skipped 2 previous similar messages [ 6588.412666] Lustre: Skipped 1 previous similar message [ 6593.054602] Lustre: server umount lustre-MDT0000 complete [ 6596.487378] LustreError: 161268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672015 with bad export cookie 3525530983374160253 [ 6596.494928] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6596.496744] LustreError: 161268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6596.902956] Lustre: server umount lustre-MDT0001 complete [ 6610.759364] Lustre: server umount lustre-OST0000 complete [ 6629.554267] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 6640.010370] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 6673.512456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.722796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6673.746195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6673.889170] Lustre: lustre-MDT0000: new disk, initializing [ 6674.024061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6674.033309] Lustre: Skipped 1 previous similar message [ 6674.071873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6679.044950] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6690.866600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.957056] Lustre: 170234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6691.258696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6696.006912] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6705.767540] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6711.541857] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6717.864774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6718.113233] Lustre: lustre-OST0000: new disk, initializing [ 6718.115488] Lustre: Skipped 1 previous similar message [ 6718.122739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6718.127636] Lustre: Skipped 2 previous similar messages [ 6718.132107] Lustre: 172200:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6723.493354] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6727.731085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6727.739490] Lustre: Skipped 1 previous similar message [ 6727.746160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6727.840174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6731.114594] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6735.050438] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6735.254718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6738.974387] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6739.223840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6743.525143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6743.532652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6743.542134] LustreError: Skipped 1 previous similar message [ 6743.543958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6743.543966] Lustre: Skipped 2 previous similar messages [ 6743.579322] Lustre: Skipped 2 previous similar messages [ 6755.295130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6755.499627] Lustre: server umount lustre-OST0000 complete [ 6762.976688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6773.215140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6773.416562] Lustre: server umount lustre-MDT0000 complete [ 6774.248682] LustreError: 170245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6774.268531] LustreError: 170245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 6777.956792] LustreError: 170226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672196 with bad export cookie 3525530983374177060 [ 6777.987216] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6778.905061] Lustre: server umount lustre-MDT0001 complete [ 6789.618534] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:23:26 (1788672206) [ 6796.314624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6801.369250] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6812.544121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6819.802299] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6828.040559] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6833.650671] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6839.836079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6846.459928] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6854.542305] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6868.763981] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:24:45 (1788672285) [ 6873.058123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.075917] Lustre: Skipped 2 previous similar messages [ 6873.085319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6873.087656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6873.091439] Lustre: Skipped 10 previous similar messages [ 6878.731859] Lustre: server umount lustre-OST0000 complete [ 6888.677311] Lustre: server umount lustre-MDT0000 complete [ 6892.450414] LustreError: 174597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672311 with bad export cookie 3525530983374177942 [ 6892.464472] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6892.469668] LustreError: 174597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6892.863733] Lustre: server umount lustre-MDT0001 complete [ 6902.610916] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 6905.098821] Key type lgssc unregistered [ 6905.360950] LNet: 178344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6905.368971] LNetError: 178344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6905.380573] LNet: Removed LNI 192.168.206.130@tcp [ 6906.062803] Key type .llcrypt unregistered [ 6906.065438] Key type ._llcrypt unregistered [ 6923.055276] Key type ._llcrypt registered [ 6923.057524] Key type .llcrypt registered [ 6923.135219] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 6924.039220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6924.072338] alg: No test for adler32 (adler32-zlib) [ 6925.073628] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 6925.268720] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 6926.919172] Key type lgssc registered [ 6928.069985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6934.888836] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6934.918352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6950.431413] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6955.551476] LustreError: 179583:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.130@tcp: failed processing log, type 4: rc = -110 [ 6960.735746] LustreError: 179583:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6960.753694] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6960.761966] LustreError: MGC192.168.206.130@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6960.773551] LustreError: 179583:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6960.780250] LustreError: 179583:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -22 [ 6960.786906] LustreError: 179583:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 6960.868638] Lustre: server umount lustre-OST0000 complete [ 6960.873806] LustreError: 179583:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6970.380722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6972.050718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6977.824697] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 6981.608586] LustreError: 180083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.622319] LustreError: 180083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6986.725371] LustreError: 180082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6987.966390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6988.382247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6993.224941] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7000.832296] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7006.601761] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7013.289194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7013.493813] LustreError: 181899:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7013.504516] Lustre: 181899:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.130@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7013.636855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7020.242704] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7028.391915] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7032.290095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7032.292753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7032.304360] Lustre: Skipped 1 previous similar message [ 7033.379026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7033.391344] Lustre: Skipped 3 previous similar messages [ 7036.937897] Lustre: server umount lustre-MDT0000 complete [ 7038.434293] LustreError: 180812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7038.454365] LustreError: 180812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7040.408251] LustreError: 180063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672459 with bad export cookie 5211647575397878185 [ 7040.419056] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7040.424269] LustreError: 180063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7041.631277] LustreError: 180814:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000550e3177 x1875558890763520/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7041.815392] Lustre: server umount lustre-MDT0001 complete [ 7056.203806] Lustre: server umount lustre-OST0000 complete [ 7065.072875] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 7074.556958] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7113.104161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7113.336792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7113.361962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7113.449669] Lustre: lustre-MDT0000: new disk, initializing [ 7113.583189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7113.619615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7118.325170] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7127.501382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7127.598184] Lustre: 185424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7127.650365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7127.653802] Lustre: Skipped 1 previous similar message [ 7127.713881] Lustre: lustre-MDT0001: new disk, initializing [ 7127.793619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7127.834597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7127.852149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7132.625814] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7141.538459] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7147.742229] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7155.392930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7155.754093] Lustre: lustre-OST0000: new disk, initializing [ 7155.769582] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7155.780995] Lustre: 187393:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7155.951035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7164.516047] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7166.091289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7166.124103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7166.356625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7175.394554] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7180.152475] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7180.375226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7185.261484] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7185.437571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7192.033152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7192.040202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7192.042954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7192.056093] Lustre: Skipped 2 previous similar messages [ 7192.068139] Lustre: Skipped 2 previous similar messages [ 7193.484697] Lustre: server umount lustre-OST0000 complete [ 7197.152593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7197.172433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7197.183881] Lustre: Skipped 1 previous similar message [ 7202.272381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7202.284263] Lustre: Skipped 1 previous similar message [ 7203.532658] Lustre: server umount lustre-MDT0000 complete [ 7207.394908] LustreError: 185433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7207.413512] LustreError: 185433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7208.029985] LustreError: 185419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672626 with bad export cookie 5211647575397878906 [ 7208.030753] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7208.047492] LustreError: 185419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7208.622633] Lustre: server umount lustre-MDT0001 complete [ 7217.598743] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:30:34 (1788672634) [ 7234.728942] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 7237.535685] Key type lgssc unregistered [ 7237.797893] LNet: 190339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7237.811729] LNetError: 190339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7237.844939] LNet: Removed LNI 192.168.206.130@tcp [ 7238.707695] Key type .llcrypt unregistered [ 7238.709564] Key type ._llcrypt unregistered [ 7254.247096] Key type ._llcrypt registered [ 7254.250084] Key type .llcrypt registered [ 7254.328631] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7255.202575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7255.217241] alg: No test for adler32 (adler32-zlib) [ 7256.190570] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 7256.421589] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 7258.097097] Key type lgssc registered [ 7259.043776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7266.273747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7266.298323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.690353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7271.593435] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7278.054056] LustreError: 191555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7278.077993] LustreError: 191555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7278.715177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.056848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7282.955768] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7289.267655] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7293.694595] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7299.906865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7300.029823] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7300.033243] LustreError: 193418:0:(llog.c:1696:llog_backup()) MGC192.168.206.130@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7300.039794] Lustre: 193418:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.130@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7300.155064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.226907] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7311.753701] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7314.912559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7314.931202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7314.949832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7317.482473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7317.483280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7317.507359] Lustre: Skipped 1 previous similar message [ 7322.593855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7322.601864] Lustre: Skipped 4 previous similar messages [ 7327.715278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7327.729790] Lustre: Skipped 2 previous similar messages [ 7328.223152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7328.464223] Lustre: server umount lustre-MDT0000 complete [ 7332.731454] LustreError: 191533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672751 with bad export cookie 15368723213919915708 [ 7332.737385] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7332.745751] LustreError: 191533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7332.832701] LustreError: 192988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7332.842238] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.876981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7333.146843] Lustre: server umount lustre-MDT0001 complete [ 7339.545414] Lustre: server umount lustre-OST0000 complete [ 7347.956585] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 7356.713822] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7389.546043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7389.735765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7389.760459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7389.818727] Lustre: lustre-MDT0000: new disk, initializing [ 7389.882890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7389.911199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7393.401964] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7401.801922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7401.928799] Lustre: 196941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7401.966168] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7401.968448] Lustre: Skipped 1 previous similar message [ 7401.970504] LustreError: 197722:0:(llog.c:1696:llog_backup()) MGC192.168.206.130@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7401.975966] LustreError: 197722:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7401.979918] Lustre: 197722:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.130@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7401.994413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7401.998608] Lustre: Skipped 1 previous similar message [ 7402.058857] Lustre: lustre-MDT0001: new disk, initializing [ 7402.160928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7402.207822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7402.225469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7405.812196] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7414.067726] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7419.321391] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7424.803795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7424.990110] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7424.993979] Lustre: Skipped 1 previous similar message [ 7425.001534] LustreError: 198882:0:(llog.c:1696:llog_backup()) MGC192.168.206.130@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7425.012674] LustreError: 198882:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7425.020449] Lustre: 198882:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.130@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7425.037265] Lustre: 198882:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7425.062838] Lustre: lustre-OST0000: new disk, initializing [ 7425.067825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7425.072151] Lustre: 198907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7425.142216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7430.604444] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7434.331350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7434.338887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7434.408856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7439.909430] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7445.091689] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7445.324468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7449.770679] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7449.970433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7455.205170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7455.205736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7455.206097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7455.216501] Lustre: Skipped 1 previous similar message [ 7455.234577] LustreError: Skipped 1 previous similar message [ 7458.129386] Lustre: server umount lustre-OST0000 complete [ 7463.905044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7463.912931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7475.684413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7475.694053] Lustre: Skipped 7 previous similar messages [ 7476.191160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7476.428829] Lustre: server umount lustre-MDT0000 complete [ 7480.708416] LustreError: 196933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672899 with bad export cookie 15368723213919916429 [ 7480.714671] LustreError: 196933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7480.716787] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7481.066491] Lustre: server umount lustre-MDT0001 complete [ 7490.122084] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:35:06 (1788672906) [ 7506.625695] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7517.870784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7518.261195] LustreError: 202035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7518.295613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7523.309995] LustreError: 202036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.098924] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7527.396647] LustreError: 202035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7534.210341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7540.132140] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7543.925837] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7551.160344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7551.543173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7551.547575] Lustre: Skipped 1 previous similar message [ 7557.743661] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7566.112543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7566.328544] Lustre: lustre-OST0001: new disk, initializing [ 7566.334580] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7566.345546] Lustre: 204342:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7572.976931] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7574.648248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7574.668546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7574.802757] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7584.883324] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7596.281365] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7598.740245] Lustre: DEBUG MARKER: ADD OST3 [ 7604.138307] loop: module loaded [ 7604.220038] loop0: detected capacity change from 0 to 204800000 [ 7604.467630] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7604.488470] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7604.498747] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7610.511638] loop0: detected capacity change from 0 to 204800000 [ 7611.094280] Lustre: DEBUG MARKER: STOP OST3 [ 7616.532472] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7616.728291] Lustre: lustre-OST0003: new disk, initializing [ 7616.737136] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7616.749097] Lustre: 206364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7616.952592] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7616.963173] Lustre: Skipped 1 previous similar message [ 7618.216704] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7618.322343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7618.334162] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7618.388097] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7620.256128] LustreError: 206506:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7620.840175] Lustre: Failing over lustre-OST0003 [ 7621.011931] Lustre: server umount lustre-OST0003 complete [ 7623.616236] LustreError: 203521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7623.646796] LustreError: 203521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7623.679042] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7623.687762] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7623.710679] Lustre: Skipped 2 previous similar messages [ 7625.177535] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7635.016169] Lustre: DEBUG MARKER: START OST3 again [ 7642.584332] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7642.914534] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7644.046314] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7644.985733] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7644.994334] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7649.454603] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7671.272287] Lustre: Failing over lustre-OST0003 [ 7671.465585] Lustre: server umount lustre-OST0003 complete [ 7673.850238] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7673.857472] LustreError: 206928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.876506] Lustre: Skipped 1 previous similar message [ 7673.893882] LustreError: 206928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7689.190753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7689.195601] Lustre: Skipped 2 previous similar messages [ 7695.214339] Lustre: server umount lustre-MDT0000 complete [ 7699.013331] LustreError: 202017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673117 with bad export cookie 15368723213919917311 [ 7699.018361] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7699.026880] LustreError: 202017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7705.408738] Lustre: server umount lustre-MDT0001 complete [ 7710.074902] Lustre: server umount lustre-OST0000 complete [ 7713.981779] Lustre: server umount lustre-OST0001 complete [ 7722.791578] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 7732.926788] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7767.295191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7767.500799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7767.537386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7767.597832] Lustre: lustre-MDT0000: new disk, initializing [ 7767.675100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7767.680847] Lustre: Skipped 1 previous similar message [ 7767.690228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7771.454974] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7782.796293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7782.979829] Lustre: 211806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7783.389377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7788.268151] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7798.896058] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7805.312719] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7811.669641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7811.937702] Lustre: lustre-OST0000: new disk, initializing [ 7811.946527] Lustre: Skipped 1 previous similar message [ 7811.952228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7811.956451] Lustre: Skipped 2 previous similar messages [ 7811.961468] Lustre: 213772:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7814.011176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7814.024957] Lustre: Skipped 1 previous similar message [ 7814.035201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7814.149351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7818.163790] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7828.304315] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7833.232839] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7833.441347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7838.618995] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7838.867375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7844.834296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7844.842890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7844.854295] Lustre: Skipped 7 previous similar messages [ 7844.856640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7844.866527] LustreError: Skipped 1 previous similar message [ 7844.882110] Lustre: Skipped 10 previous similar messages [ 7847.658865] Lustre: server umount lustre-OST0000 complete [ 7855.074246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7858.493614] Lustre: server umount lustre-MDT0000 complete [ 7860.198366] LustreError: 211813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7860.227646] LustreError: 211813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 7862.366183] LustreError: 211797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673281 with bad export cookie 15368723213919919607 [ 7862.375945] LustreError: 211797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7862.379044] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7862.769971] Lustre: server umount lustre-MDT0001 complete [ 7866.974230] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:41:23 (1788673283) [ 7883.081593] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 7892.530200] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 7929.492112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7929.803561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7929.845579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7929.943651] Lustre: lustre-MDT0000: new disk, initializing [ 7930.070144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7935.126783] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7946.601677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7946.800816] Lustre: 218385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7947.214731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7951.578964] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7960.944620] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7967.440954] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7974.775863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7975.140305] Lustre: 220352:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7977.106669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7981.968961] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 7992.534778] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7997.480836] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7997.766965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8002.136770] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8002.345948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8007.649237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8007.653078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8007.673150] Lustre: Skipped 2 previous similar messages [ 8007.674550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8007.710327] Lustre: Skipped 3 previous similar messages [ 8010.570932] Lustre: server umount lustre-OST0000 complete [ 8021.013979] Lustre: server umount lustre-MDT0000 complete [ 8023.008595] LustreError: 218390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8023.022097] LustreError: 218390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8024.631897] LustreError: 219180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673443 with bad export cookie 15368723213919920489 [ 8024.635684] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8024.658861] LustreError: 219180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8025.148446] Lustre: server umount lustre-MDT0001 complete [ 8033.540863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.239057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8034.251647] Lustre: Skipped 5 previous similar messages [ 8039.403762] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8047.688508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8052.327361] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8060.523485] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8067.319807] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8074.235358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8081.068531] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8088.970714] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8099.300161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8113.119947] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8113.351753] Lustre: server umount lustre-OST0000 complete [ 8123.349221] Lustre: server umount lustre-MDT0000 complete [ 8126.842922] LustreError: 222552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673545 with bad export cookie 15368723213919921371 [ 8126.851604] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8126.854809] LustreError: 222552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8127.297299] Lustre: server umount lustre-MDT0001 complete [ 8225.010610] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 8228.354273] Key type lgssc unregistered [ 8228.728548] LNet: 226231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8228.741172] LNetError: 226231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8228.793610] LNet: Removed LNI 192.168.206.130@tcp [ 8229.555156] Key type .llcrypt unregistered [ 8229.558398] Key type ._llcrypt unregistered [ 8260.674218] Key type ._llcrypt registered [ 8260.678941] Key type .llcrypt registered [ 8260.835282] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 8275.487709] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 8276.247728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8276.385753] alg: No test for adler32 (adler32-zlib) [ 8277.463480] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 8277.667339] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 8279.335179] Key type lgssc registered [ 8280.291278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8310.474828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8310.504346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8311.686380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8311.714298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8311.795223] Lustre: lustre-MDT0000: new disk, initializing [ 8311.930882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8311.953887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8315.174551] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8324.342275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8324.481451] Lustre: 229581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8324.551505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8324.573044] Lustre: Skipped 1 previous similar message [ 8324.701171] Lustre: lustre-MDT0001: new disk, initializing [ 8324.840173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8324.907797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8324.917915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8329.417333] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8338.846157] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8343.917362] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8350.270428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8350.491679] Lustre: lustre-OST0000: new disk, initializing [ 8350.501198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8350.507143] Lustre: 231548:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8350.615505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8356.186246] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8359.512354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8359.521978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8359.630109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8365.544526] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8370.535990] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8370.736616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8375.648970] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8375.823864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8380.386108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8380.390521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.400581] LustreError: Skipped 1 previous similar message [ 8380.408387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8380.416532] Lustre: Skipped 1 previous similar message [ 8380.440541] Lustre: Skipped 1 previous similar message [ 8383.939079] Lustre: server umount lustre-OST0000 complete [ 8390.625303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8390.632930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8393.996801] Lustre: server umount lustre-MDT0000 complete [ 8397.541265] LustreError: 229574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673816 with bad export cookie 13686028398276434957 [ 8397.543284] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8397.559963] LustreError: 229574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8397.982526] Lustre: server umount lustre-MDT0001 complete [ 8415.380390] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 8424.708198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.998613] LustreError: 234483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8425.016378] LustreError: 234483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8425.045592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8428.607491] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8430.052993] LustreError: 234484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8430.066624] LustreError: 234484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8435.180173] LustreError: 234483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8436.758934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.126157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8441.990890] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8445.248705] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8452.708037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8453.054307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8458.665049] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8466.371861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8466.526385] Lustre: lustre-OST0001: new disk, initializing [ 8466.529967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8466.534457] Lustre: 236787:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8472.720320] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8475.273678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8475.289449] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8475.362918] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8484.474907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8491.798944] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8504.543972] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:52:01 (1788673921) [ 8508.896943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8508.906163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8508.920909] Lustre: Skipped 1 previous similar message [ 8508.927380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8508.936908] Lustre: Skipped 1 previous similar message [ 8511.457630] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8511.461212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8511.478218] Lustre: Skipped 1 previous similar message [ 8511.509022] Lustre: Skipped 3 previous similar messages [ 8516.578875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8516.586323] Lustre: Skipped 3 previous similar messages [ 8521.183530] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8521.382961] Lustre: server umount lustre-MDT0000 complete [ 8521.701993] LustreError: 234480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.727444] LustreError: 234480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8524.854286] LustreError: 237614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673943 with bad export cookie 13686028398276435839 [ 8524.860647] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8524.875810] LustreError: 237614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8525.314700] Lustre: server umount lustre-MDT0001 complete [ 8539.250128] Lustre: server umount lustre-OST0000 complete [ 8553.233940] Lustre: server umount lustre-OST0001 complete [ 8559.334481] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 8569.027160] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 8605.195996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.348126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8605.367635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8605.428959] Lustre: lustre-MDT0000: new disk, initializing [ 8605.525297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8605.527994] Lustre: Skipped 1 previous similar message [ 8605.543369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8611.107913] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8621.988945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8622.165954] Lustre: 240737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8622.229621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8622.234415] Lustre: Skipped 1 previous similar message [ 8622.388333] Lustre: lustre-MDT0001: new disk, initializing [ 8622.538278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8622.557788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8627.225501] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8637.395978] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8642.705775] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8650.553830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8650.942501] Lustre: lustre-OST0000: new disk, initializing [ 8650.951557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8650.962122] Lustre: 242707:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8651.099289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8651.108882] Lustre: Skipped 1 previous similar message [ 8652.511810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8652.528296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8652.681052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8658.243308] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8668.709473] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8673.380755] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8673.536574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8678.117884] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8678.437258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8683.490125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8683.504629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8683.506548] Lustre: Skipped 1 previous similar message [ 8683.514400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8683.516155] LustreError: Skipped 1 previous similar message [ 8683.531628] Lustre: Skipped 1 previous similar message [ 8686.381621] Lustre: server umount lustre-OST0000 complete [ 8693.730783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8693.747959] Lustre: Skipped 1 previous similar message [ 8696.645676] Lustre: server umount lustre-MDT0000 complete [ 8698.851424] LustreError: 240744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8698.883478] LustreError: 240744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8700.379697] LustreError: 242706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674119 with bad export cookie 13686028398276437974 [ 8700.390121] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8700.395078] LustreError: 242706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8700.785568] Lustre: server umount lustre-MDT0001 complete [ 8709.319986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8709.884545] LustreError: 244923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.452422] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8722.382647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8722.626352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8722.630555] Lustre: Skipped 1 previous similar message [ 8726.416628] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8733.479462] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8738.542596] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8745.167380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8750.909351] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 8757.369200] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8903.661873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8903.664562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8903.664940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8903.664945] Lustre: Skipped 2 previous similar messages [ 8903.677837] Lustre: Skipped 2 previous similar messages [ 8908.222534] Lustre: server umount lustre-OST0000 complete [ 8927.199386] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8927.368961] Lustre: server umount lustre-MDT0000 complete [ 8929.252682] LustreError: 246890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8929.274630] LustreError: 246890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8931.414831] LustreError: 244903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674350 with bad export cookie 13686028398276438856 [ 8931.426362] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8931.435239] LustreError: 244903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8931.803156] Lustre: server umount lustre-MDT0001 complete [ 9023.785747] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [ 9026.035854] Key type lgssc unregistered [ 9026.297804] LNet: 248530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9026.302705] LNetError: 248530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9026.325628] LNet: Removed LNI 192.168.206.130@tcp [ 9027.080495] Key type .llcrypt unregistered [ 9027.086149] Key type ._llcrypt unregistered [ 9051.189543] Key type ._llcrypt registered [ 9051.191830] Key type .llcrypt registered [ 9051.298069] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 9065.713115] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9066.502824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9066.609850] alg: No test for adler32 (adler32-zlib) [ 9067.706169] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 9067.989157] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [ 9069.744355] Key type lgssc registered [ 9070.835421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9112.045319] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9123.184604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9123.214419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9124.481640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9124.512622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9124.590612] Lustre: lustre-MDT0000: new disk, initializing [ 9124.638656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9124.651251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9129.212647] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9138.963306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9139.117782] Lustre: 252604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9139.178722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9139.182926] Lustre: Skipped 1 previous similar message [ 9139.259528] Lustre: lustre-MDT0001: new disk, initializing [ 9139.344171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9139.386448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9139.406627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9143.530151] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9148.077850] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9154.143352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9154.332500] Lustre: lustre-OST0000: new disk, initializing [ 9154.335535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9154.340546] Lustre: 254236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9154.421334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9160.084180] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9161.801904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9161.814849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9161.904470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9170.703628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9170.911467] Lustre: lustre-OST0001: new disk, initializing [ 9170.917390] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9170.926111] Lustre: 255106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9171.045322] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9177.414736] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9177.690460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9177.703094] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9177.771028] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9187.661893] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9194.539414] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9206.240658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9206.246279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9206.258185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9208.804974] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9208.810339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9208.824144] Lustre: Skipped 2 previous similar messages [ 9208.847244] Lustre: Skipped 3 previous similar messages [ 9210.595687] Lustre: server umount lustre-MDT0000 complete [ 9214.043291] LustreError: 252595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674632 with bad export cookie 11914517521421744653 [ 9214.047680] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9214.058660] LustreError: 252595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9214.390452] Lustre: server umount lustre-MDT0001 complete [ 9228.359806] Lustre: server umount lustre-OST0000 complete [ 9242.493401] Lustre: server umount lustre-OST0001 complete [ 9256.220548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9265.287205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9274.644300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9285.251634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9305.225543] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9314.957458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9314.979169] Lustre: lustre-MDT0000: reset Object Index mappings [ 9315.484943] LustreError: 260121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9315.506157] LustreError: 260121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9315.566061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9320.445623] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9320.931763] LustreError: 260122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9326.063969] LustreError: 260121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9329.057631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9329.085081] Lustre: lustre-MDT0001: reset Object Index mappings [ 9329.540380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9332.820541] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9335.116657] Lustre: 261261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9340.327582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9341.610532] LustreError: 261617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9341.628455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9345.583254] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9347.041349] LustreError: 261617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.803373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9354.074873] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9354.080732] Lustre: Skipped 1 previous similar message [ 9355.138269] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9359.231918] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9366.322279] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9369.433840] Lustre: 263133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9375.712231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9375.721595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9375.728705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9375.737871] Lustre: Skipped 2 previous similar messages [ 9375.747325] Lustre: Skipped 1 previous similar message [ 9376.274488] Lustre: server umount lustre-MDT0000 complete [ 9379.693559] LustreError: 260102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674798 with bad export cookie 11914517521421751443 [ 9379.704194] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9379.719336] LustreError: 260102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9380.048554] Lustre: server umount lustre-MDT0001 complete [ 9393.777772] Lustre: server umount lustre-OST0000 complete [ 9407.100789] Lustre: server umount lustre-OST0001 complete [ 9424.920478] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 02:07:21 (1788674841) [ 9440.765940] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 9450.497574] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9491.609400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9491.812490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9491.830557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9491.885682] Lustre: lustre-MDT0000: new disk, initializing [ 9491.941090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.964299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9495.837967] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9504.487101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9504.595668] Lustre: 267390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9504.634241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9504.639277] Lustre: Skipped 1 previous similar message [ 9504.717365] Lustre: lustre-MDT0001: new disk, initializing [ 9504.823311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9504.832626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9508.384773] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9516.348591] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9521.827970] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9528.181635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9528.417592] Lustre: lustre-OST0000: new disk, initializing [ 9528.421184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9528.425623] Lustre: 269359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9529.989218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9529.995470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9530.109558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9534.422422] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9542.599400] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9546.514309] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9546.767486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9551.537185] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9551.810906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9555.943460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9555.943959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9555.952195] LustreError: Skipped 1 previous similar message [ 9555.957399] Lustre: Skipped 1 previous similar message [ 9555.962573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9555.971471] Lustre: Skipped 3 previous similar messages [ 9559.713663] Lustre: server umount lustre-OST0000 complete [ 9566.177433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9566.185110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9566.193787] Lustre: Skipped 2 previous similar messages [ 9566.202838] Lustre: Skipped 1 previous similar message [ 9569.344079] Lustre: server umount lustre-MDT0000 complete [ 9571.296146] LustreError: 267401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9571.328249] LustreError: 267401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9572.611393] LustreError: 268186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674991 with bad export cookie 11914517521421753536 [ 9572.624352] LustreError: 268186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9572.632828] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9572.990566] Lustre: server umount lustre-MDT0001 complete [ 9590.399768] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9599.974921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9600.501510] LustreError: 272293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9600.549177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9600.555722] Lustre: Skipped 2 previous similar messages [ 9604.962708] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9613.028561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9617.098409] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9619.480675] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9625.591727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9630.935894] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9637.356025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9637.585147] Lustre: lustre-OST0001: new disk, initializing [ 9637.588458] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9637.595555] Lustre: 274598:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9639.249194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9639.260672] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9639.381925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9643.381621] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9652.515119] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9656.026645] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9660.897945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9660.931925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9669.602846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9669.617796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9669.636581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9674.753918] Lustre: server umount lustre-MDT0000 complete [ 9675.235872] LustreError: 273022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9675.251455] LustreError: 273022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9677.964785] LustreError: 272273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675096 with bad export cookie 11914517521421754418 [ 9677.969713] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9677.985654] LustreError: 272273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9678.266017] Lustre: server umount lustre-MDT0001 complete [ 9691.763204] Lustre: server umount lustre-OST0000 complete [ 9705.497438] Lustre: server umount lustre-OST0001 complete [ 9719.289252] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9729.098912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9729.642369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9729.647315] Lustre: Skipped 3 previous similar messages [ 9734.013674] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9739.758883] LustreError: 277939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9739.772601] LustreError: 277939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9742.607590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9747.384488] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9750.157369] Lustre: 279080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9757.152926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9763.613600] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9772.792200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9775.146995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9775.305811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9781.957601] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9789.472955] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9793.144973] Lustre: 280957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9798.638695] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9798.710736] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9813.139084] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 02:13:49 (1788675229) [ 9829.858264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9829.869714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9829.883577] Lustre: Skipped 3 previous similar messages [ 9829.890364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9829.895725] Lustre: Skipped 4 previous similar messages [ 9834.457799] Lustre: server umount lustre-MDT0000 complete [ 9838.546853] LustreError: 279083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675257 with bad export cookie 11914517521421760795 [ 9838.554098] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9838.554718] LustreError: 279083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9845.018261] Lustre: server umount lustre-MDT0001 complete [ 9855.444242] Lustre: server umount lustre-OST0000 complete [ 9859.542891] Lustre: server umount lustre-OST0001 complete [ 9866.509584] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [ 9876.040491] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9926.791398] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [ 9936.692270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9936.909438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9936.978639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9937.103051] Lustre: lustre-MDT0000: new disk, initializing [ 9937.207275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9941.671780] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9953.681027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9953.800956] Lustre: 285485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9954.184026] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9960.092320] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9973.211212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9973.516420] Lustre: 287120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9975.470158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9980.105422] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [ 9991.235486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9991.442598] Lustre: 287991:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9991.578756] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9991.589488] Lustre: Skipped 6 previous similar messages [ 9993.385544] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9993.400310] Lustre: Skipped 1 previous similar message [ 9993.597233] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9998.001924] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10008.100808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10012.354489] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10012.365924] Lustre: Skipped 1 previous similar message [10027.144198] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:17:24 (1788675444) [10029.543940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10029.551839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10029.562105] Lustre: Skipped 8 previous similar messages [10029.577060] Lustre: Skipped 11 previous similar messages [10031.984471] Lustre: server umount lustre-MDT0000 complete [10034.674477] LustreError: 285498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10034.699465] LustreError: 285498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [10035.900831] LustreError: 287119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675454 with bad export cookie 11914517521421768719 [10035.904377] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10035.911630] LustreError: 287119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10036.256515] Lustre: server umount lustre-MDT0001 complete [10050.934180] Lustre: server umount lustre-OST0000 complete [10064.893555] Lustre: server umount lustre-OST0001 complete [10074.030694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10078.074622] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10086.564402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.570781] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10098.585893] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10104.281238] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10111.403542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10118.006547] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10125.673933] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10150.584167] loop1: detected capacity change from 0 to 204800000 [10150.840630] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10150.851338] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10150.865860] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10156.331525] loop1: detected capacity change from 0 to 204800000 [10162.710907] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10162.799474] LustreError: 290225:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs6268: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10162.833276] LustreError: 290225:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs6268 [10162.839388] LustreError: 290225:0:(mgs_handler.c:517:mgs_target_reg()) fs6268-MDT0001: Error checking target index: rc = -13 [10162.849761] LustreError: fs6268-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10162.866587] LustreError: 293980:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -13 [10162.876628] LustreError: 293980:0:(tgt_mount.c:2134:server_put_super()) no obd fs6268-MDT0001 [10162.883029] LustreError: 293980:0:(tgt_mount.c:116:server_deregister_mount()) fs6268-MDT0001 not registered [10162.927422] Lustre: server umount fs6268-MDT0001 complete [10162.929793] LustreError: 293980:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10171.805293] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10172.030423] Lustre: Setting parameter fs6268-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs6268-MDT0001 [10172.106786] Lustre: srv-fs6268-MDT0001: No data found on store. Initialize space. [10172.120605] Lustre: Skipped 4 previous similar messages [10172.247313] Lustre: fs6268-MDT0001: new disk, initializing [10172.252786] Lustre: Skipped 3 previous similar messages [10172.423081] Lustre: srv-fs6268-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10172.429967] Lustre: 294448:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6268-MDT0001-mdtlov: can't create new llog: rc = -115 [10172.441772] LustreError: 294448:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6268-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10178.213534] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10186.263595] Lustre: server umount fs6268-MDT0001 complete [10197.990607] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10200.303551] loop1: detected capacity change from 0 to 204800000 [10206.707389] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10207.238505] Lustre: 296015:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6268-MDT0001-mdtlov: can't create new llog: rc = -115 [10207.253418] LustreError: 296015:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6268-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10211.588172] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10216.574442] Lustre: server umount fs6268-MDT0001 complete [10228.641884] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.30@tcp (stopping) [10228.717397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10228.735156] Lustre: Skipped 1 previous similar message [10233.923580] Lustre: server umount lustre-MDT0000 complete [10242.916498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10243.158836] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10247.899907] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10248.694321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10256.158663] loop1: detected capacity change from 0 to 204800000 [10263.411044] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.210659] Lustre: 298461:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs6268-MDT0001-mdtlov: can't create new llog: rc = -115 [10264.226208] LustreError: 298461:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs6268-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10270.579486] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10276.791151] Lustre: server umount fs6268-MDT0001 complete [10294.769204] LustreError: 292068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10294.787975] LustreError: 292068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 109 previous similar messages [10305.001893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10305.011246] LustreError: Skipped 1 previous similar message [10308.590926] Lustre: server umount lustre-OST0000 complete [10318.546405] Lustre: server umount lustre-MDT0000 complete [10322.647219] LustreError: 290217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675741 with bad export cookie 11914517521421771407 [10322.662218] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10322.682344] LustreError: 290217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10323.318534] Lustre: server umount lustre-MDT0001 complete [10336.019899] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [10339.543854] Key type lgssc unregistered [10339.833602] LNet: 300805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10339.842383] LNetError: 300805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10339.859991] LNet: Removed LNI 192.168.206.130@tcp [10340.755550] Key type .llcrypt unregistered [10340.760358] Key type ._llcrypt unregistered [10361.825177] Key type ._llcrypt registered [10361.829537] Key type .llcrypt registered [10361.956207] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [10362.983868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10363.134040] alg: No test for adler32 (adler32-zlib) [10364.167626] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [10364.352359] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [10366.007184] Key type lgssc registered [10367.133464] Lustre: Echo OBD driver; http://www.lustre.org/ [10374.719895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10374.737636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10376.112644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10380.684586] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10385.383621] LustreError: 302072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10385.421127] LustreError: 302072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10390.505276] LustreError: 302073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10390.719558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10391.136281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10396.039186] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10404.782048] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10411.230282] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10417.589653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10417.945854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10421.043345] LustreError: 303902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.349247] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10426.352430] LustreError: 303903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10431.468947] LustreError: 303904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10432.655982] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10440.593749] LustreError: 303904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10440.616123] LustreError: 303904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10453.762247] loop1: detected capacity change from 0 to 204800000 [10453.939993] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10453.968984] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10453.997714] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10459.420641] loop1: detected capacity change from 0 to 204800000 [10461.079458] LustreError: 303904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10461.100790] LustreError: 303904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10464.790811] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10464.953726] Lustre: fs23215-OST0000: new disk, initializing [10464.958959] Lustre: srv-fs23215-OST0000: No data found on store. Initialize space. [10464.969571] Lustre: 305809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10465.055599] Lustre: fs23215-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10472.402347] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10480.987960] Lustre: server umount fs23215-OST0000 complete [10492.733263] loop1: detected capacity change from 0 to 204800000 [10496.919698] LustreError: 303903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10496.938732] LustreError: 303903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [10499.110270] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10499.428305] Lustre: fs23215-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10502.131261] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10505.866974] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10515.805489] loop2: detected capacity change from 0 to 204800000 [10516.039934] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10516.057506] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10516.069206] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10517.489319] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10521.441946] loop2: detected capacity change from 0 to 204800000 [10526.712817] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10526.824483] LustreError: fs23215-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10526.839692] LustreError: fs23215-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10526.854278] LustreError: 308847:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -13 [10526.867221] LustreError: 308847:0:(tgt_mount.c:2134:server_put_super()) no obd fs23215-OST0001 [10526.878440] LustreError: 308847:0:(tgt_mount.c:116:server_deregister_mount()) fs23215-OST0001 not registered [10526.922728] Lustre: server umount fs23215-OST0001 complete [10526.926247] LustreError: 308847:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10537.281967] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10537.422183] Lustre: fs23215-OST0001: new disk, initializing [10537.438432] Lustre: srv-fs23215-OST0001: No data found on store. Initialize space. [10537.454561] Lustre: 309400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10537.537758] Lustre: fs23215-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10544.298971] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10552.085394] Lustre: server umount fs23215-OST0000 complete [10560.590376] Lustre: server umount fs23215-OST0001 complete [10561.423111] LustreError: 303904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.448221] LustreError: 303904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10571.992186] loop1: detected capacity change from 0 to 204800000 [10577.653687] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10577.837391] Lustre: fs23215-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10584.400575] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10592.260282] loop2: detected capacity change from 0 to 204800000 [10600.136959] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10609.491960] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10620.405611] Lustre: server umount fs23215-OST0000 complete [10630.273125] Lustre: server umount fs23215-OST0001 complete [10645.482150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10645.491696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10645.503184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10645.507656] Lustre: Skipped 1 previous similar message [10645.519637] Lustre: Skipped 1 previous similar message [10648.311603] Lustre: server umount lustre-OST0000 complete [10652.128315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10652.137733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10652.157801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10655.713387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10655.713470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10658.188423] Lustre: server umount lustre-MDT0000 complete [10662.459261] LustreError: 302052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676081 with bad export cookie 143900912196978054 [10662.466189] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10662.482546] LustreError: 302052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10662.892847] Lustre: server umount lustre-MDT0001 complete [10676.816310] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [10680.381456] Key type lgssc unregistered [10680.741304] LNet: 315322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10680.754049] LNetError: 315322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10681.832672] LNet: Removed LNI 192.168.206.130@tcp [10682.916178] Key type .llcrypt unregistered [10682.919918] Key type ._llcrypt unregistered [10715.534140] Key type ._llcrypt registered [10715.536668] Key type .llcrypt registered [10715.698260] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [10729.920414] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [10730.693179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10730.751149] alg: No test for adler32 (adler32-zlib) [10731.744678] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [10731.940280] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [10733.623159] Key type lgssc registered [10734.638483] Lustre: Echo OBD driver; http://www.lustre.org/ [10772.343936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10772.380234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10773.709111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10773.749384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10773.852386] Lustre: lustre-MDT0000: new disk, initializing [10773.938129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10773.958940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10778.222266] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10788.036889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10788.161217] Lustre: 319182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10788.260100] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10788.262928] Lustre: Skipped 1 previous similar message [10788.421355] Lustre: lustre-MDT0001: new disk, initializing [10788.552948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10788.633799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10788.643196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10792.465583] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10801.015760] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10806.737893] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10813.807142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10814.166928] Lustre: lustre-OST0000: new disk, initializing [10814.175213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10814.179901] Lustre: 321150:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10814.298472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10819.677106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10819.690254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10819.803530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10821.508590] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [10831.098194] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10833.333978] Lustre: Mounted lustre-client - version 2.17.58_3_gb7bcc87 [10834.248914] Lustre: Unmounted lustre-client [10843.624400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10843.626310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10843.637722] Lustre: Skipped 1 previous similar message [10848.746241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10848.752724] Lustre: Skipped 2 previous similar messages [10849.006816] Lustre: server umount lustre-OST0000 complete [10853.858187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10853.870365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10853.874258] Lustre: Skipped 1 previous similar message [10858.899353] Lustre: server umount lustre-MDT0000 complete [10862.856972] LustreError: 319976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676281 with bad export cookie 2831037769428872240 [10862.860090] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10862.866282] LustreError: 319976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10863.217294] Lustre: server umount lustre-MDT0001 complete [10872.513485] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [10874.883059] Key type lgssc unregistered [10888.479832] LNet: 323310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10888.484816] LNetError: 323310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10962.223481] LNet: Removed LNI 192.168.206.130@tcp [10963.080685] Key type .llcrypt unregistered [10963.083182] Key type ._llcrypt unregistered [10979.477276] Key type ._llcrypt registered [10979.478743] Key type .llcrypt registered [10979.572795] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [10993.144522] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [10993.970923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10994.114439] alg: No test for adler32 (adler32-zlib) [10995.129793] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [10995.337443] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [10997.039185] Key type lgssc registered [10998.120581] Lustre: Echo OBD driver; http://www.lustre.org/ [11027.700746] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:34:04 (1788676444) [11033.907025] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11033.951842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11035.166118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11035.200712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11035.284583] Lustre: lustre-MDT0000: new disk, initializing [11035.355608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11035.381550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11039.582383] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11049.886906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.044660] Lustre: 326526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11050.133967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11050.138026] Lustre: Skipped 1 previous similar message [11050.190871] Lustre: lustre-MDT0001: new disk, initializing [11050.264875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11050.302194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11050.315307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11054.510964] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11063.195914] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11068.621872] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11075.574629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11075.844442] Lustre: lustre-OST0000: new disk, initializing [11075.849586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11075.854995] Lustre: 328496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11075.942738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11081.891970] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11084.322489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11084.333885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11084.417422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11091.013286] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11104.743374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.749778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11104.756414] Lustre: Skipped 1 previous similar message [11109.860748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11109.865863] Lustre: Skipped 2 previous similar messages [11110.064916] Lustre: server umount lustre-OST0000 complete [11114.977207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11114.997495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11115.002530] Lustre: Skipped 1 previous similar message [11119.905127] Lustre: server umount lustre-MDT0000 complete [11123.553767] LustreError: 326520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676542 with bad export cookie 16565275628068107123 [11123.558295] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11123.560543] LustreError: 326520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11123.815348] Lustre: server umount lustre-MDT0001 complete [11133.207641] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11135.768841] Key type lgssc unregistered [11136.078476] LNet: 330404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11136.084526] LNetError: 330404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11136.117495] LNet: Removed LNI 192.168.206.130@tcp [11136.917094] Key type .llcrypt unregistered [11136.921387] Key type ._llcrypt unregistered [11156.816675] Key type ._llcrypt registered [11156.818539] Key type .llcrypt registered [11156.922963] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [11158.063312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11158.146869] alg: No test for adler32 (adler32-zlib) [11159.229917] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [11159.445546] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [11161.119176] Key type lgssc registered [11162.176905] Lustre: Echo OBD driver; http://www.lustre.org/ [11169.775779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11169.789590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11171.306769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11176.014075] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11180.515375] LustreError: 331675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11180.533663] LustreError: 331675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11183.781128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11184.119704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11188.640859] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11195.482921] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11201.467652] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11208.521554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11208.747683] LustreError: 333493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11209.058699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11216.399565] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11227.035232] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11238.508952] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11240.114918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11245.476399] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.30@tcp (stopping) [11245.536580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11245.539107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11246.050770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11246.053274] Lustre: Skipped 1 previous similar message [11246.054670] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11246.070509] Lustre: Skipped 1 previous similar message [11250.570390] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.30@tcp (stopping) [11250.580995] Lustre: Skipped 1 previous similar message [11255.686669] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.30@tcp (stopping) [11255.693754] Lustre: Skipped 3 previous similar messages [11258.338375] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11258.607661] Lustre: server umount lustre-MDT0000 complete [11260.808667] LustreError: 331669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11262.041513] LustreError: 331656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676680 with bad export cookie 13164878182290743021 [11262.045926] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11262.049778] LustreError: 331656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11265.929168] LustreError: 331671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11265.943480] LustreError: 331671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11265.966199] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.30@tcp (stopping) [11265.980126] Lustre: Skipped 3 previous similar messages [11266.542483] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11268.683901] Lustre: server umount lustre-MDT0001 complete [11272.439967] Lustre: server umount lustre-OST0000 complete [11280.460975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11284.260128] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11286.813122] Lustre: 335855:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.130@tcp [11288.760220] Lustre: server umount MGS complete [11296.975638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11297.336169] LustreError: 336429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11297.351525] LustreError: 336429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11297.398398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11302.313276] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11305.880321] LustreError: 336424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11305.910252] LustreError: 336424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11312.177104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11312.521579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11317.325923] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11326.058708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11326.425401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11332.112551] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11339.382844] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11341.610139] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11348.835461] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11350.943835] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11358.181832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11358.189911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11358.191609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11358.201849] Lustre: Skipped 1 previous similar message [11358.244788] Lustre: Skipped 3 previous similar messages [11364.170692] Lustre: server umount lustre-OST0000 complete [11368.422388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11368.441518] Lustre: Skipped 1 previous similar message [11378.664088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11378.674801] Lustre: Skipped 6 previous similar messages [11382.239830] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11382.488709] Lustre: server umount lustre-MDT0000 complete [11383.777370] LustreError: 337882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11383.796073] LustreError: 337882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11386.169640] LustreError: 336411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676804 with bad export cookie 13164878182290743861 [11386.177987] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11386.180557] LustreError: 336411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11386.594903] Lustre: server umount lustre-MDT0001 complete [11396.927816] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11399.595889] Key type lgssc unregistered [11399.955728] LNet: 339865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11399.963430] LNetError: 339865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11399.987199] LNet: Removed LNI 192.168.206.130@tcp [11400.725982] Key type .llcrypt unregistered [11400.732580] Key type ._llcrypt unregistered [11418.991411] Key type ._llcrypt registered [11418.994470] Key type .llcrypt registered [11419.129580] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [11434.775806] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [11436.016220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11436.050507] alg: No test for adler32 (adler32-zlib) [11437.052035] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [11437.327145] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [11439.031424] Key type lgssc registered [11440.299523] Lustre: Echo OBD driver; http://www.lustre.org/ [11475.955982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11475.987487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11477.252620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11477.292981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11477.413609] Lustre: lustre-MDT0000: new disk, initializing [11477.529459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11477.557850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11482.569822] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11493.062257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11493.183808] Lustre: 343024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11493.266240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11493.272553] Lustre: Skipped 1 previous similar message [11493.363571] Lustre: lustre-MDT0001: new disk, initializing [11493.456194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11493.494796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11493.502040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11498.137612] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11507.616816] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11513.244664] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11519.638153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11519.916291] Lustre: lustre-OST0000: new disk, initializing [11519.921507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11519.932436] Lustre: 344994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11520.007927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11526.408538] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11528.760614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11528.766140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11528.862071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11535.070691] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11539.545762] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11539.770286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11543.963263] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11544.208847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11549.154856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11549.161661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11549.181766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11549.664811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11549.674160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11549.699189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11552.154187] Lustre: server umount lustre-OST0000 complete [11559.907128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11559.915415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11559.918233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11559.927873] Lustre: Skipped 1 previous similar message [11561.989552] Lustre: server umount lustre-MDT0000 complete [11566.320249] LustreError: 343018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676985 with bad export cookie 10425161651007056033 [11566.324698] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11566.330252] LustreError: 343018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11566.736917] Lustre: server umount lustre-MDT0001 complete [11579.209448] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:43:15 (1788676995) [11597.551750] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [11605.718953] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [11636.432642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11636.660620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11636.690922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11636.745905] Lustre: lustre-MDT0000: new disk, initializing [11636.829693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11636.844175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11641.166187] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11650.984114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11651.079493] Lustre: 349753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11651.115954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11651.118801] Lustre: Skipped 1 previous similar message [11651.174845] Lustre: lustre-MDT0001: new disk, initializing [11651.262587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11651.290015] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11651.308859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11655.497309] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11665.034285] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11670.728802] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11677.656626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11678.025238] Lustre: lustre-OST0000: new disk, initializing [11678.033843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11678.039055] Lustre: 351718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11678.136981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11684.461941] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [11685.473297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11685.483351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11685.618415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11693.691839] Lustre: DEBUG MARKER: oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11731.955187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11731.968489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11731.983777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11732.011802] Lustre: Skipped 2 previous similar messages [11737.058247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11737.065699] Lustre: Skipped 1 previous similar message [11742.175146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11742.346634] Lustre: server umount lustre-OST0000 complete [11747.299068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11747.309617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11747.315052] Lustre: Skipped 2 previous similar messages [11747.317436] Lustre: Skipped 2 previous similar messages [11752.425306] LustreError: 349764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11752.440717] LustreError: 349764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11752.510617] Lustre: server umount lustre-MDT0000 complete [11756.237974] LustreError: 349745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677174 with bad export cookie 10425161651007056915 [11756.259785] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11756.598877] Lustre: server umount lustre-MDT0001 complete [11766.690576] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11769.249524] Key type lgssc unregistered [11769.471733] LNet: 353893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11769.477424] LNetError: 353893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11770.542048] LNet: Removed LNI 192.168.206.130@tcp [11771.410605] Key type .llcrypt unregistered [11771.412958] Key type ._llcrypt unregistered [11853.972430] Key type ._llcrypt registered [11853.977326] Key type .llcrypt registered [11854.088711] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11854.345502] Key type .llcrypt unregistered [11854.350034] Key type ._llcrypt unregistered [11923.119949] Key type ._llcrypt registered [11923.121756] Key type .llcrypt registered [11923.226239] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11923.513415] Key type .llcrypt unregistered [11923.515320] Key type ._llcrypt unregistered [11951.558744] Key type ._llcrypt registered [11951.560853] Key type .llcrypt registered [11951.658588] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11951.934781] Key type .llcrypt unregistered [11951.939401] Key type ._llcrypt unregistered [11976.459496] Key type ._llcrypt registered [11976.462072] Key type .llcrypt registered [11976.538628] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [11976.844361] Key type .llcrypt unregistered [11976.847168] Key type ._llcrypt unregistered [12043.962986] Key type ._llcrypt registered [12043.966184] Key type .llcrypt registered [12044.071239] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12044.319296] Key type .llcrypt unregistered [12044.321584] Key type ._llcrypt unregistered [12071.113207] Key type ._llcrypt registered [12071.114850] Key type .llcrypt registered [12071.204882] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12071.453292] Key type .llcrypt unregistered [12071.456306] Key type ._llcrypt unregistered [12096.402446] Key type ._llcrypt registered [12096.405982] Key type .llcrypt registered [12096.489859] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12096.719493] Key type .llcrypt unregistered [12096.721548] Key type ._llcrypt unregistered [12127.711885] Key type ._llcrypt registered [12127.715848] Key type .llcrypt registered [12127.823425] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12128.181736] Key type .llcrypt unregistered [12128.189035] Key type ._llcrypt unregistered [12156.510663] Key type ._llcrypt registered [12156.512937] Key type .llcrypt registered [12156.609272] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12156.890406] Key type .llcrypt unregistered [12156.894317] Key type ._llcrypt unregistered [12185.195236] Key type ._llcrypt registered [12185.200929] Key type .llcrypt registered [12185.301458] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12185.516347] Key type .llcrypt unregistered [12185.518936] Key type ._llcrypt unregistered [12234.905325] Key type ._llcrypt registered [12234.907406] Key type .llcrypt registered [12235.020702] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12235.309831] Key type .llcrypt unregistered [12235.317509] Key type ._llcrypt unregistered [12258.882329] Key type ._llcrypt registered [12258.885346] Key type .llcrypt registered [12259.006929] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12259.273183] Key type .llcrypt unregistered [12259.277854] Key type ._llcrypt unregistered [12286.421855] Key type ._llcrypt registered [12286.427686] Key type .llcrypt registered [12286.528346] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12286.775335] Key type .llcrypt unregistered [12286.777381] Key type ._llcrypt unregistered [12336.765517] Key type ._llcrypt registered [12336.767830] Key type .llcrypt registered [12336.877936] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [12350.738044] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [12351.660356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12351.667710] alg: No test for adler32 (adler32-zlib) [12352.730521] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [12352.955805] LNet: Added LNI 192.168.206.130@tcp [8/256/0/180] [12354.575188] Key type lgssc registered [12355.773877] Lustre: Echo OBD driver; http://www.lustre.org/ [12388.745186] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12388.777420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12390.046855] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12390.081757] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12390.162795] Lustre: tst250fs-MDT0000: new disk, initializing [12390.264908] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12390.302667] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12394.989307] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [12406.008940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12406.200533] Lustre: 374039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12406.296939] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12406.306144] Lustre: Skipped 1 previous similar message [12406.435418] Lustre: tst250fs-MDT0001: new disk, initializing [12406.581622] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12406.678846] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12406.692748] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12411.425949] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [12416.177997] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12422.102129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12422.446238] Lustre: tst250fs-OST0000: new disk, initializing [12422.451204] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12422.461241] Lustre: 375670:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12422.571762] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12428.872098] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12428.888816] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12428.994831] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12429.019673] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [12440.296956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12440.434376] Lustre: tst250fs-OST0001: new disk, initializing [12440.438131] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12440.443417] Lustre: 376541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12440.523125] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12445.822713] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12445.835366] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12445.941047] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12447.633675] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_default_debug -1 all [12453.403422] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12456.034902] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12456.038537] Lustre: Skipped 2 previous similar messages [12473.312160] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12473.322362] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12473.340749] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12476.899982] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12476.902314] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12476.923710] Lustre: Skipped 1 previous similar message [12478.697817] Lustre: server umount tst250fs-MDT0000 complete [12481.843754] LustreError: 374029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677900 with bad export cookie 10657777329203342658 [12481.852665] LustreError: MGC192.168.206.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12481.858027] LustreError: 374029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12482.023059] Lustre: tst250fs-MDT0001-lwp-OST0000: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12482.030503] Lustre: Skipped 2 previous similar messages [12482.148372] Lustre: server umount tst250fs-MDT0001 complete [12492.175063] Lustre: server umount tst250fs-OST0000 complete [12495.591952] Lustre: server umount tst250fs-OST0001 complete [12511.212568] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [12518.841918] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [12545.991988] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:59:23 (1788677963) [12547.017963] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12561.728089] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing set_hostid [12571.531389] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing load_modules_local [12599.234505] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12361 sec ========= 03:00:15 (1788678015) [12600.708175] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:00:17 (1788678017) === [12603.359088] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:00:20 (1788678020) === [12637.616506] Lustre: DEBUG MARKER: oleg630-server.virtnet: executing unload_modules_local [12640.327987] Key type lgssc unregistered [12640.616155] LNet: 385662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12640.627720] LNetError: 385662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12640.656047] LNet: Removed LNI 192.168.206.130@tcp [12641.631298] Key type .llcrypt unregistered [12641.633887] Key type ._llcrypt unregistered