[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 501278356 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003406] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008573] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010016] pid_max: default: 32768 minimum: 301 [ 0.011147] LSM: Security Framework initializing [ 0.012069] Yama: becoming mindful. [ 0.014049] SELinux: Initializing. [ 0.015109] *** VALIDATE selinux *** [ 0.026119] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032067] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035148] *** VALIDATE tmpfs *** [ 0.036520] *** VALIDATE proc *** [ 0.038120] *** VALIDATE cgroup *** [ 0.039010] *** VALIDATE cgroup2 *** [ 0.040324] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041191] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043033] Spectre V2 : User space: Vulnerable [ 0.044009] Speculative Store Bypass: Vulnerable [ 0.047409] debug: unmapping init [mem 0xffffffffb0a59000-0xffffffffb0a60fff] [ 0.050231] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051889] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052024] ... version: 2 [ 0.053014] ... bit width: 48 [ 0.054013] ... generic registers: 4 [ 0.055013] ... value mask: 0000ffffffffffff [ 0.056013] ... max period: 00007fffffffffff [ 0.057013] ... fixed-purpose events: 3 [ 0.058014] ... event mask: 000000070000000f [ 0.059312] rcu: Hierarchical SRCU implementation. [ 0.061416] smp: Bringing up secondary CPUs ... [ 0.062588] x86: Booting SMP configuration: [ 0.063033] .... node #0, CPUs: #1 #2 #3 [ 0.066356] smp: Brought up 1 node, 4 CPUs [ 0.068012] smpboot: Max logical packages: 1 [ 0.069019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149015] node 0 deferred pages initialised in 75ms [ 0.153013] devtmpfs: initialized [ 0.154167] x86/mm: Memory block size: 128MB [ 0.156816] gcov: version magic: 0x41383552 [ 0.157590] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161139] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163355] pinctrl core: initialized pinctrl subsystem [ 0.165165] [ 0.165745] ************************************************************* [ 0.168013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171013] ** ** [ 0.173011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174010] ** ** [ 0.177013] ** This means that this kernel is built to expose internal ** [ 0.179012] ** IOMMU data structures, which may compromise security on ** [ 0.182012] ** your system. ** [ 0.184011] ** ** [ 0.186014] ** If you see this message and you are not debugging the ** [ 0.188013] ** kernel, report this immediately to your vendor! ** [ 0.190012] ** ** [ 0.192010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194010] ************************************************************* [ 0.196654] NET: Registered protocol family 16 [ 0.198479] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202107] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205373] cpuidle: using governor menu [ 0.206590] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209485] PCI: Using configuration type 1 for base access [ 0.212128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220121] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224092] cryptd: max_cpu_qlen set to 1000 [ 0.227244] ACPI: Added _OSI(Module Device) [ 0.228013] ACPI: Added _OSI(Processor Device) [ 0.229029] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236833] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242582] ACPI: Interpreter enabled [ 0.244077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246019] ACPI: Using IOAPIC for interrupt routing [ 0.248140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.251458] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263475] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268186] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.271103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276758] acpiphp: Slot [2] registered [ 0.279232] acpiphp: Slot [5] registered [ 0.280172] acpiphp: Slot [6] registered [ 0.282200] acpiphp: Slot [7] registered [ 0.284108] acpiphp: Slot [8] registered [ 0.285137] acpiphp: Slot [9] registered [ 0.287190] acpiphp: Slot [10] registered [ 0.288123] acpiphp: Slot [3] registered [ 0.291121] acpiphp: Slot [4] registered [ 0.292107] acpiphp: Slot [11] registered [ 0.294136] acpiphp: Slot [12] registered [ 0.296375] acpiphp: Slot [13] registered [ 0.297106] acpiphp: Slot [14] registered [ 0.299115] acpiphp: Slot [15] registered [ 0.301117] acpiphp: Slot [16] registered [ 0.303100] acpiphp: Slot [17] registered [ 0.304121] acpiphp: Slot [18] registered [ 0.305099] acpiphp: Slot [19] registered [ 0.307100] acpiphp: Slot [20] registered [ 0.308105] acpiphp: Slot [21] registered [ 0.310091] acpiphp: Slot [22] registered [ 0.311153] acpiphp: Slot [23] registered [ 0.313117] acpiphp: Slot [24] registered [ 0.314158] acpiphp: Slot [25] registered [ 0.315108] acpiphp: Slot [26] registered [ 0.317106] acpiphp: Slot [27] registered [ 0.318106] acpiphp: Slot [28] registered [ 0.320104] acpiphp: Slot [29] registered [ 0.321114] acpiphp: Slot [30] registered [ 0.322110] acpiphp: Slot [31] registered [ 0.324086] PCI host bridge to bus 0000:00 [ 0.325019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.329033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.334035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.336066] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338288] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342264] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.346907] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357790] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.363686] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.371027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375569] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.377757] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.381826] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.399015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408517] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.417023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.455022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.468749] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.475017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.481015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.501019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.509377] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.532026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.558023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.590021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.601000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.612014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.619015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.639014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.651673] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.659024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.672075] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.698018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.706772] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.719018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.728015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.753016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.765460] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.768072] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.775407] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.781393] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.788268] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.798079] iommu: Default domain type: Passthrough [ 0.807273] SCSI subsystem initialized [ 0.811166] ACPI: bus type USB registered [ 0.813200] usbcore: registered new interface driver usbfs [ 0.817078] usbcore: registered new interface driver hub [ 0.819090] usbcore: registered new device driver usb [ 0.824180] pps_core: LinuxPPS API ver. 1 registered [ 0.828021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.832061] PTP clock support registered [ 0.836353] EDAC MC: Ver: 3.0.0 [ 0.837000] PCI: Using ACPI for IRQ routing [ 0.841076] NetLabel: Initializing [ 0.844016] NetLabel: domain hash size = 128 [ 0.848012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.851096] NetLabel: unlabeled traffic allowed by default [ 0.853626] vgaarb: loaded [ 0.856041] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.859014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.868122] clocksource: Switched to clocksource kvm-clock [ 1.248437] VFS: Disk quotas dquot_6.6.0 [ 1.254149] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.265417] *** VALIDATE ramfs *** [ 1.268170] *** VALIDATE hugetlbfs *** [ 1.272437] pnp: PnP ACPI init [ 1.296132] pnp: PnP ACPI: found 6 devices [ 1.373075] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.380858] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.383126] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.388787] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.399379] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.404033] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.415084] NET: Registered protocol family 2 [ 1.419886] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.428198] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.441036] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.455091] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.459535] TCP: Hash tables configured (established 65536 bind 65536) [ 1.471547] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.476598] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.480724] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.485283] NET: Registered protocol family 1 [ 1.493541] RPC: Registered named UNIX socket transport module. [ 1.499382] RPC: Registered udp transport module. [ 1.501719] RPC: Registered tcp transport module. [ 1.503409] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.518480] NET: Registered protocol family 44 [ 1.526048] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.531326] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.538201] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.546231] PCI: CLS 0 bytes, default 64 [ 1.553329] Unpacking initramfs... [ 5.825389] debug: unmapping init [mem 0xffff9678bcc54000-0xffff9678bffbffff] [ 5.830352] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.836408] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.844687] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.541686] Initialise system trusted keyrings [ 7.547295] Key type blacklist registered [ 7.558031] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.580511] zbud: loaded [ 7.590494] *** VALIDATE nfs *** [ 7.592679] *** VALIDATE nfs4 *** [ 7.596190] pstore: using deflate compression [ 7.606439] Platform Keyring initialized [ 7.901808] NET: Registered protocol family 38 [ 7.904446] Key type asymmetric registered [ 7.908538] Asymmetric key parser 'x509' registered [ 7.920948] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.930936] io scheduler mq-deadline registered [ 7.933854] io scheduler kyber registered [ 7.938506] io scheduler bfq registered [ 7.950645] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.956621] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.961967] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.970463] ACPI: Power Button [PWRF] [ 7.990969] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.005036] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 8.036398] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 8.046436] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.081667] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.112542] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.149126] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.162145] Non-volatile memory driver v1.3 [ 8.170170] Linux agpgart interface v0.103 [ 8.241548] virtio_blk virtio1: [vda] 146008 512-byte logical blocks (74.8 MB/71.3 MiB) [ 8.245271] vda: detected capacity change from 0 to 74756096 [ 8.284826] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.289182] vdb: detected capacity change from 0 to 1073741824 [ 8.323802] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.329474] vdc: detected capacity change from 0 to 2621440000 [ 8.356713] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.362931] vdd: detected capacity change from 0 to 2621440000 [ 8.395788] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.401887] vde: detected capacity change from 0 to 4294967296 [ 8.427270] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.430827] vdf: detected capacity change from 0 to 4294967296 [ 8.439745] libphy: Fixed MDIO Bus: probed [ 8.449887] usbcore: registered new interface driver usbserial_generic [ 8.452482] usbserial: USB Serial support registered for generic [ 8.454580] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.459385] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.461386] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.464636] mousedev: PS/2 mouse device common for all mice [ 8.469597] rtc_cmos 00:05: RTC can wake from S4 [ 8.471152] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.473131] rtc_cmos 00:05: registered as rtc0 [ 8.473184] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.473227] intel_pstate: CPU model not supported [ 8.477644] hid: raw HID events driver (C) Jiri Kosina [ 8.489225] usbcore: registered new interface driver usbhid [ 8.492486] usbhid: USB HID core driver [ 8.494494] drop_monitor: Initializing network drop monitor service [ 8.497852] Initializing XFRM netlink socket [ 8.499382] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.500987] NET: Registered protocol family 10 [ 8.506668] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.506911] Segment Routing with IPv6 [ 8.510117] NET: Registered protocol family 17 [ 8.512902] mpls_gso: MPLS GSO support [ 8.519413] RAS: Correctable Errors collector initialized. [ 8.521372] AVX version of gcm_enc/dec engaged. [ 8.523472] AES CTR mode by8 optimization enabled [ 8.615783] sched_clock: Marking stable (8615702513, 0)->(9572677915, -956975402) [ 8.620492] registered taskstats version 1 [ 8.622936] Loading compiled-in X.509 certificates [ 8.626144] zswap: loaded using pool lzo/zbud [ 8.657967] Key type big_key registered [ 8.676440] Key type encrypted registered [ 8.678699] ima: No TPM chip found, activating TPM-bypass! [ 8.683948] ima: Allocated hash algorithm: sha1 [ 8.685642] ima: No architecture policies found [ 8.687497] evm: Initialising EVM extended attributes: [ 8.689808] evm: security.selinux [ 8.691172] evm: security.ima [ 8.692377] evm: security.capability [ 8.694214] evm: HMAC attrs: 0x1 [ 8.697512] rtc_cmos 00:05: setting system clock to 2026-08-22 08:24:43 UTC (1787387083) [ 8.703852] debug: unmapping init [mem 0xffffffffb1a03000-0xffffffffb1bfffff] [ 8.707872] debug: unmapping init [mem 0xffffffffb0782000-0xffffffffb0a58fff] [ 8.717065] Write protecting the kernel read-only data: 28672k [ 8.723185] debug: unmapping init [mem 0xffffffffaee03000-0xffffffffaeffffff] [ 8.727357] debug: unmapping init [mem 0xffffffffaf714000-0xffffffffaf7fffff] [ 8.771520] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.795037] systemd[1]: Detected virtualization kvm. [ 8.802298] systemd[1]: Detected architecture x86-64. [ 8.811334] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.892169] systemd[1]: No hostname configured. [ 8.894434] systemd[1]: Set hostname to . [ 8.897435] random: systemd: uninitialized urandom read (16 bytes read) [ 8.907526] systemd[1]: Initializing machine ID from random generator. [ 9.284548] random: systemd: uninitialized urandom read (16 bytes read) [ 9.290445] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 9.301603] random: systemd: uninitialized urandom read (16 bytes read) [ 9.306436] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 9.318980] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel.[ 9.552912] random: fast init done [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.586562] device-mapper: uevent: version 1.0.3 [ 10.589801] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 12.808856] virtio_net virtio0 ens2: renamed from eth0 [ 13.005069] scsi host0: ata_piix [ 13.071780] scsi host1: ata_piix [ 13.080785] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.088871] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.207942] random: crng init done [ 18.212065] random: 7 urandom warning(s) missed due to ratelimiting [ 18.700276] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.005331] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.400567] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.000873] SELinux: Disabled at runtime. [ 24.123492] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.139599] systemd[1]: Detected virtualization kvm. [ 24.141767] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.388235] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.394416] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.409887] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.420081] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.426896] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.453414] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.469368] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 25.562831] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.866295] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.856065] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.963371] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.645178] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.741826] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 34.903751] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 35.952767] NFS: Registering the id_resolver key type [ 35.958085] Key type id_resolver registered [ 35.961297] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg612-server login: [ 93.116467] libcfs: loading out-of-tree module taints kernel. [ 93.150525] Key type ._llcrypt registered [ 93.152805] Key type .llcrypt registered [ 93.269616] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 111.119410] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 113.285345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 113.303687] alg: No test for adler32 (adler32-zlib) [ 114.797532] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 116.027710] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 117.776352] Key type lgssc registered [ 119.801980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 136.620436] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.420378] hrtimer: interrupt took 7690543 ns [ 178.726989] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 192.837920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.898814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.205305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 194.235731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.334435] Lustre: lustre-MDT0000: new disk, initializing [ 194.434781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.453380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.178222] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.012793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.124173] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.153150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 213.159179] Lustre: Skipped 1 previous similar message [ 213.271382] Lustre: lustre-MDT0001: new disk, initializing [ 213.386776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.449552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.465420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.187996] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.463598] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 231.263678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.550525] Lustre: lustre-OST0000: new disk, initializing [ 231.553643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.560403] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.635256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.903450] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.220571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 241.231444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 241.316621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 250.228422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.381698] Lustre: lustre-OST0001: new disk, initializing [ 250.384930] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 250.390055] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.466126] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.066802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 251.086674] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 251.143912] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.640698] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.194528] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 280.127389] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 286.815667] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing check_logdir /tmp/testlogs/ [ 292.287711] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing yml_node [ 297.149869] Lustre: DEBUG MARKER: Client: 2.17.57.45 [ 299.459079] Lustre: DEBUG MARKER: MDS: 2.17.57.45 [ 301.497079] Lustre: DEBUG MARKER: OSS: 2.17.57.45 [ 302.905951] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 04:29:36 EDT 2026 [ 316.837114] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 318.018741] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.286023] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.295503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.301122] Lustre: Skipped 3 previous similar messages [ 333.319638] Lustre: Skipped 3 previous similar messages [ 338.402981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.413448] Lustre: Skipped 3 previous similar messages [ 339.388397] Lustre: server umount lustre-MDT0000 complete [ 342.627647] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387417 with bad export cookie 17032588984889950025 [ 342.628162] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.638519] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 343.522116] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.548667] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.553467] Lustre: Skipped 1 previous similar message [ 347.559756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 347.565509] Lustre: Skipped 1 previous similar message [ 349.166301] Lustre: server umount lustre-MDT0001 complete [ 353.305440] Lustre: server umount lustre-OST0000 complete [ 357.081122] Lustre: server umount lustre-OST0001 complete [ 363.693232] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 373.651226] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 406.758790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.026292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 407.048289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 407.123815] Lustre: lustre-MDT0000: new disk, initializing [ 407.234506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.265609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 411.562548] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 421.129854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.224370] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 421.272652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 421.275181] Lustre: Skipped 1 previous similar message [ 421.334504] Lustre: lustre-MDT0001: new disk, initializing [ 421.414789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.438953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.453490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.626106] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 433.746426] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 440.141462] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.418141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.668229] Lustre: lustre-OST0000: new disk, initializing [ 447.672474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 447.678329] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 447.781743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.284308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.294151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.424411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 454.873583] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 464.023325] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 469.216660] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 469.389283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 474.270760] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 474.462413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.226581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.230751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.241201] Lustre: Skipped 2 previous similar messages [ 480.254255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.268440] Lustre: Skipped 1 previous similar message [ 482.751461] Lustre: server umount lustre-OST0000 complete [ 487.905527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 487.910430] LustreError: Skipped 1 previous similar message [ 487.912656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.465500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 490.470309] Lustre: Skipped 1 previous similar message [ 493.403815] Lustre: server umount lustre-MDT0000 complete [ 495.593293] LustreError: 15086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.616591] LustreError: 15086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 497.600592] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387572 with bad export cookie 17032588984889951558 [ 497.601494] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.619611] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 498.238129] Lustre: server umount lustre-MDT0001 complete [ 504.019245] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:32:57 (1787387577) [ 524.123338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.556795] LustreError: 19921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.670918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.340669] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 529.894077] LustreError: 19922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.013427] LustreError: 19921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.414754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.995097] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 547.849303] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 552.351887] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 554.355718] Lustre: Failing over lustre-MDT0000 [ 554.468574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.471089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.479142] Lustre: Skipped 2 previous similar messages [ 554.489168] Lustre: Skipped 1 previous similar message [ 554.615711] Lustre: server umount lustre-MDT0000 complete [ 559.592974] LustreError: 19918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.614135] LustreError: 19918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 571.881857] LustreError: 20972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.897541] LustreError: 20972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 572.776165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.898825] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 576.839111] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 578.531855] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 578.542897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 578.560762] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 585.815124] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 589.669288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.676049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 589.688176] Lustre: Skipped 1 previous similar message [ 589.702245] Lustre: Skipped 1 previous similar message [ 596.013862] Lustre: server umount lustre-MDT0000 complete [ 599.751394] LustreError: 19902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387674 with bad export cookie 17032588984889952972 [ 599.761517] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.768696] LustreError: 19902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 600.213174] Lustre: server umount lustre-MDT0001 complete [ 608.107044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.611237] LustreError: 23258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.712739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.718876] Lustre: Skipped 2 previous similar messages [ 613.342394] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 622.519277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.695707] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 634.327345] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 639.556931] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 641.550900] Lustre: Failing over lustre-MDT0000 [ 641.717644] Lustre: server umount lustre-MDT0000 complete [ 643.555287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.558681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 643.568282] Lustre: Skipped 1 previous similar message [ 643.570419] LustreError: 23258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.605428] LustreError: 23258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 659.938581] Lustre: 3649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387718/real 1787387718] req@ffff967803458380 x1874211310323968/t0(0) o400->MGC192.168.206.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787387734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 659.959756] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.985392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.177939] LustreError: 3646:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff967803937b80 x1874211310327040/t0(0) o250->MGC192.168.206.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 670.537488] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 675.613748] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 675.809497] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 675.828110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 675.834572] Lustre: Skipped 1 previous similar message [ 675.849456] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 686.091309] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 690.281481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.283958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 690.287403] Lustre: Skipped 1 previous similar message [ 690.293995] Lustre: Skipped 3 previous similar messages [ 696.661620] Lustre: server umount lustre-MDT0000 complete [ 700.253695] LustreError: 23239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387775 with bad export cookie 17032588984889953910 [ 700.256334] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 700.265296] LustreError: 23239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 700.544875] Lustre: server umount lustre-MDT0001 complete [ 709.003941] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:36:21 (1787387781) [ 723.675728] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 734.067328] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 771.245172] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 780.715531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.899415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 780.919397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 780.988932] Lustre: lustre-MDT0000: new disk, initializing [ 781.064599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 781.069148] Lustre: Skipped 2 previous similar messages [ 781.080260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 784.569939] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 793.837281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.985638] Lustre: 29628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 794.300808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 798.276560] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 805.985940] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 811.722420] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 818.870774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.135547] Lustre: 31597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 821.171586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 821.225534] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 821.261463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 821.274482] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 825.855277] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 833.479230] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 838.026360] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 857.064296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.070153] Lustre: Skipped 1 previous similar message [ 857.070637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 857.073566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 857.078400] Lustre: Skipped 3 previous similar messages [ 857.091298] LustreError: Skipped 1 previous similar message [ 861.282797] Lustre: server umount lustre-OST0000 complete [ 866.289278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 871.104933] Lustre: server umount lustre-MDT0000 complete [ 872.421670] LustreError: 29641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 872.441349] LustreError: 29641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 874.635830] LustreError: 29621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387949 with bad export cookie 17032588984889954316 [ 874.642935] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 874.649526] LustreError: 29621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 875.163989] Lustre: server umount lustre-MDT0001 complete [ 885.435424] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 888.234729] Key type lgssc unregistered [ 888.466463] LNet: 33578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.480411] LNetError: 33578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.507617] LNet: Removed LNI 192.168.206.112@tcp [ 889.311503] Key type .llcrypt unregistered [ 889.315846] Key type ._llcrypt unregistered [ 909.661059] Key type ._llcrypt registered [ 909.662721] Key type .llcrypt registered [ 909.766580] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 910.885110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 910.915097] alg: No test for adler32 (adler32-zlib) [ 911.918675] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 912.151623] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 913.856538] Key type lgssc registered [ 914.895824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 921.987937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 922.005135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.555315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 927.857180] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 933.871686] LustreError: 34973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 933.895026] LustreError: 34973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 936.393873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 936.728739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 942.457305] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 950.353733] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 954.623535] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 960.634364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 961.274224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.300785] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 967.349363] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 969.478973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 974.331854] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 990.244566] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:41:03 (1787388063) [ 992.740033] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 995.175814] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1003.629210] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:41:16 (1787388076) [ 1012.488499] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:41:25 (1787388085) [ 1013.385843] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1021.996768] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1022.004601] Lustre: Skipped 2 previous similar messages [ 1023.665402] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:41:37 (1787388097) [ 1027.554930] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1027.562369] Lustre: Skipped 3 previous similar messages [ 1045.917028] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1045.922540] Lustre: Skipped 6 previous similar messages [ 1048.407188] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:42:01 (1787388121) [ 1051.338736] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1055.022391] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1064.183549] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:42:17 (1787388137) [ 1065.925621] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1065.935265] Lustre: Skipped 3 previous similar messages [ 1078.439898] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:42:31 (1787388151) [ 1092.580251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1092.590772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.607970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1092.613413] Lustre: Skipped 1 previous similar message [ 1097.698052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1097.712327] Lustre: Skipped 1 previous similar message [ 1102.821889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1102.832899] Lustre: Skipped 1 previous similar message [ 1105.376133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1105.571708] Lustre: server umount lustre-OST0000 complete [ 1111.010064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1111.023993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.050134] Lustre: Skipped 1 previous similar message [ 1111.064496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1113.058328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.666128] Lustre: server umount lustre-MDT0000 complete [ 1118.180857] LustreError: 36866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.196197] LustreError: 36866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1119.265416] LustreError: 34953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388194 with bad export cookie 16792494552805550105 [ 1119.269100] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1119.302421] LustreError: 34953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1119.658689] Lustre: server umount lustre-MDT0001 complete [ 1131.327121] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1134.353830] Key type lgssc unregistered [ 1134.609281] LNet: 42819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.630616] LNetError: 42819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.664620] LNet: Removed LNI 192.168.206.112@tcp [ 1135.568535] Key type .llcrypt unregistered [ 1135.574746] Key type ._llcrypt unregistered [ 1157.206143] Key type ._llcrypt registered [ 1157.207849] Key type .llcrypt registered [ 1157.316930] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 1174.452410] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1175.251654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1175.322247] alg: No test for adler32 (adler32-zlib) [ 1176.587929] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1176.909827] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1178.624478] Key type lgssc registered [ 1179.835781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1214.970811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1215.027316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.252736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1216.274444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1216.350456] Lustre: lustre-MDT0000: new disk, initializing [ 1216.451856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.471309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1220.647795] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1230.821212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.968157] Lustre: 45971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1231.009830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1231.016418] Lustre: Skipped 1 previous similar message [ 1231.074345] Lustre: lustre-MDT0001: new disk, initializing [ 1231.216840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1231.264641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1231.277134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1235.219497] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1244.238803] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1249.953294] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1256.868680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1257.173616] Lustre: lustre-OST0000: new disk, initializing [ 1257.181534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1257.194392] Lustre: 47940:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1257.325319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.396743] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1265.750544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1265.763056] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1265.841337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1272.995049] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1277.482441] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1277.650820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1281.844659] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1282.142718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1286.630254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.631626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1286.636343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1286.659654] Lustre: Skipped 1 previous similar message [ 1286.692297] LustreError: Skipped 1 previous similar message [ 1290.528734] Lustre: server umount lustre-OST0000 complete [ 1296.877149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.912183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.926198] Lustre: Skipped 2 previous similar messages [ 1301.107943] Lustre: server umount lustre-MDT0000 complete [ 1305.737268] LustreError: 45962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388380 with bad export cookie 12061771781030910985 [ 1305.751241] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1305.760314] LustreError: 45962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1306.208691] Lustre: server umount lustre-MDT0001 complete [ 1314.970614] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:46:28 (1787388388) [ 1322.551573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1322.995694] LustreError: 50404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.002552] LustreError: 50404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1323.050698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1327.002124] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1328.098336] LustreError: 50405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.129831] LustreError: 50405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1333.227439] LustreError: 50404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.497879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1336.970757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1341.889194] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1348.314580] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1353.625605] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1359.750854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1360.202737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.318706] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1373.690544] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1378.277078] LustreError: 52933:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1379.209221] Lustre: Modifying parameter general.timeout=1 in log params [ 1380.212574] Lustre: Modifying parameter general.timeout=2 in log params [ 1382.414178] Lustre: Modifying parameter general.timeout=5 in log params [ 1382.417524] Lustre: Skipped 2 previous similar messages [ 1386.416825] Lustre: Modifying parameter general.timeout=10 in log params [ 1386.420606] Lustre: Skipped 4 previous similar messages [ 1394.591191] Lustre: Modifying parameter general.timeout=20 in log params [ 1394.596569] Lustre: Skipped 9 previous similar messages [ 1411.097877] Lustre: Modifying parameter general.timeout=39 in log params [ 1411.102724] Lustre: Skipped 18 previous similar messages [ 1443.732854] Lustre: Modifying parameter general.timeout=75 in log params [ 1443.741397] Lustre: Skipped 35 previous similar messages [ 1506.372163] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 04:49:39 (1787388579) [ 1508.372056] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1508.386322] Lustre: Skipped 56 previous similar messages [ 1509.142723] Lustre: 59877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1510.956771] Lustre: 59976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1514.571650] Lustre: 60177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1514.586591] Lustre: 60177:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1542.373949] Lustre: 61745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1542.379422] Lustre: 61745:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1542.390691] LustreError: 61745:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1546.153902] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 04:50:19 (1787388619) [ 1559.016830] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1559.026579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1559.029337] Lustre: Skipped 2 previous similar messages [ 1563.325135] Lustre: server umount lustre-MDT0000 complete [ 1564.130435] LustreError: 50400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.147776] LustreError: 50400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1566.546713] LustreError: 59632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388641 with bad export cookie 12061771781030911867 [ 1566.554637] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1566.567099] LustreError: 59632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1566.881824] Lustre: server umount lustre-MDT0001 complete [ 1580.417075] Lustre: server umount lustre-OST0000 complete [ 1604.972755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.280478] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1605.548236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.306531] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1619.525149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1619.711414] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1619.766118] Lustre: 64278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1619.887706] LustreError: 64288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1624.673963] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1630.501644] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1635.703329] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1642.132721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1642.331241] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1642.648092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.652217] Lustre: Skipped 1 previous similar message [ 1648.649479] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1655.793580] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1657.327258] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1657.333244] Lustre: Skipped 21 previous similar messages [ 1657.364285] Lustre: 66844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1666.535801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1666.539730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.544771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1666.544778] Lustre: Skipped 2 previous similar messages [ 1666.550574] LustreError: Skipped 1 previous similar message [ 1666.584696] Lustre: Skipped 2 previous similar messages [ 1669.774251] Lustre: server umount lustre-OST0000 complete [ 1676.257515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1676.266616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.283439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1676.293972] Lustre: Skipped 1 previous similar message [ 1679.123403] Lustre: server umount lustre-MDT0000 complete [ 1681.890494] LustreError: 65028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1681.900393] LustreError: 65028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1682.197509] LustreError: 64268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388756 with bad export cookie 12061771781030914933 [ 1682.200246] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1682.210238] LustreError: 64268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1682.459245] Lustre: server umount lustre-MDT0001 complete [ 1691.347179] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1694.004723] Key type lgssc unregistered [ 1694.265850] LNet: 68058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.282799] LNetError: 68058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.317223] LNet: Removed LNI 192.168.206.112@tcp [ 1695.140871] Key type .llcrypt unregistered [ 1695.146654] Key type ._llcrypt unregistered [ 1718.312824] Key type ._llcrypt registered [ 1718.315417] Key type .llcrypt registered [ 1718.400544] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1719.222622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1719.274680] alg: No test for adler32 (adler32-zlib) [ 1720.342835] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1720.531717] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1722.240198] Key type lgssc registered [ 1723.280573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1730.347207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1730.371091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.804120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.305175] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1741.292896] LustreError: 69472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.314247] LustreError: 69472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1741.934441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.328890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1745.569750] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1752.101274] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1757.177917] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1762.585433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1763.048128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.176186] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1775.467907] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1785.864042] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1797.097772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.106297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.111779] Lustre: Skipped 2 previous similar messages [ 1797.122131] Lustre: Skipped 2 previous similar messages [ 1799.998364] Lustre: server umount lustre-MDT0000 complete [ 1802.223217] LustreError: 69468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1802.244146] LustreError: 69468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1803.475168] LustreError: 69454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388878 with bad export cookie 5591718466070211416 [ 1803.487614] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1803.489125] LustreError: 69454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1803.805495] Lustre: server umount lustre-MDT0001 complete [ 1818.361564] Lustre: server umount lustre-OST0000 complete [ 1843.444452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.676492] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1843.712633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1843.999687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.336471] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1853.696281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1853.801564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1853.825814] Lustre: 74341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1854.189639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1857.473929] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1863.213931] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1868.145663] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1874.047337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1874.221754] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1874.658109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.394886] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1886.057908] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1887.479358] Lustre: 76913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1889.165343] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1896.419282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1896.426458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1896.441280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1896.936720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1899.201689] Lustre: server umount lustre-OST0000 complete [ 1905.121224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1905.130925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.141171] Lustre: Skipped 1 previous similar message [ 1905.147131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.151156] Lustre: Skipped 1 previous similar message [ 1907.172731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.174521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.592618] Lustre: server umount lustre-MDT0000 complete [ 1911.257099] LustreError: 74333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388986 with bad export cookie 5591718466070212249 [ 1911.259698] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.265468] LustreError: 74333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1911.538382] Lustre: server umount lustre-MDT0001 complete [ 1920.075354] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1922.348925] Key type lgssc unregistered [ 1922.562544] LNet: 78128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.580333] LNetError: 78128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1923.624343] LNet: Removed LNI 192.168.206.112@tcp [ 1924.430274] Key type .llcrypt unregistered [ 1924.433256] Key type ._llcrypt unregistered [ 1947.293563] Key type ._llcrypt registered [ 1947.296553] Key type .llcrypt registered [ 1947.405199] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1948.367791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1948.410805] alg: No test for adler32 (adler32-zlib) [ 1949.468836] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1949.696387] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1951.376168] Key type lgssc registered [ 1952.506524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1960.269048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1960.313581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.880858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1966.067244] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1972.195738] LustreError: 79541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.212264] LustreError: 79541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1974.333043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.007811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1979.226173] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1986.249762] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1991.679393] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1997.849560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1998.379909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.326558] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2011.273325] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2017.165729] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2017.918761] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2019.349377] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2019.353772] Lustre: Skipped 1 previous similar message [ 2021.684934] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2021.687483] Lustre: Skipped 2 previous similar messages [ 2026.518535] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2026.521620] Lustre: Skipped 5 previous similar messages [ 2034.753727] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2034.759475] Lustre: Skipped 9 previous similar messages [ 2050.957895] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2050.962393] Lustre: Skipped 18 previous similar messages [ 2083.003819] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2083.011482] Lustre: Skipped 41 previous similar messages [ 2101.746573] Lustre: 87634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2103.869586] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 04:59:37 (1787389177) [ 2105.139562] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2106.793877] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 04:59:40 (1787389180) [ 2111.971068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2111.976262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.986513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2114.019571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2114.028124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.028433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2116.449874] Lustre: server umount lustre-OST0000 complete [ 2123.244633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2123.251821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.268609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.279165] Lustre: Skipped 1 previous similar message [ 2125.961608] Lustre: server umount lustre-MDT0000 complete [ 2129.378501] LustreError: 80597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.410484] LustreError: 80597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2129.605661] LustreError: 79522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389204 with bad export cookie 8762088565984538074 [ 2129.609904] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2129.626091] LustreError: 79522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2129.985907] Lustre: server umount lustre-MDT0001 complete [ 2140.560330] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2144.275573] Key type lgssc unregistered [ 2144.620149] LNet: 88885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.626395] LNetError: 88885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.651354] LNet: Removed LNI 192.168.206.112@tcp [ 2145.449180] Key type .llcrypt unregistered [ 2145.450589] Key type ._llcrypt unregistered [ 2149.380355] Key type ._llcrypt registered [ 2149.382322] Key type .llcrypt registered [ 2149.497511] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2153.697595] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules [ 2155.033980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2155.051123] alg: No test for adler32 (adler32-zlib) [ 2156.155692] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2156.458248] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2158.112162] Key type lgssc registered [ 2159.429992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2160.268338] LustreError: 90087:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2176.205825] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2181.678030] LustreError: 90087:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2184.049422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2184.064218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.548283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.468712] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2195.942264] LustreError: 91111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.959837] LustreError: 91111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2196.784684] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:01:10 (1787389270) [ 2200.848971] LustreError: 91143:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2200.861896] LustreError: 91143:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2200.866751] LustreError: 91143:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 2201.232959] Lustre: server umount lustre-MDT0000 complete [ 2212.420135] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2214.628954] Key type lgssc unregistered [ 2214.875747] LNet: 92652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.882745] LNetError: 92652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.897140] LNet: Removed LNI 192.168.206.112@tcp [ 2215.560662] Key type .llcrypt unregistered [ 2215.565675] Key type ._llcrypt unregistered [ 2229.737602] Key type ._llcrypt registered [ 2229.740390] Key type .llcrypt registered [ 2229.813878] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2230.624426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2230.649158] alg: No test for adler32 (adler32-zlib) [ 2231.709448] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2231.950182] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2233.600422] Key type lgssc registered [ 2234.709707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2241.158832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2241.171572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.603813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.120701] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2252.794060] LustreError: 93863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.811948] LustreError: 93863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2253.530959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.156094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2258.525570] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2264.354349] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2268.785165] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2274.477495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2275.046209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2280.371869] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2286.469562] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2334.594926] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:03:27 (1787389407) [ 2336.419347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2338.177870] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:03:31 (1787389411) [ 2343.907212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.913474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2343.918470] Lustre: Skipped 1 previous similar message [ 2343.929684] Lustre: Skipped 2 previous similar messages [ 2347.869153] Lustre: server umount lustre-MDT0000 complete [ 2349.026532] LustreError: 93863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.037882] LustreError: 93863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2350.655625] LustreError: 93843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389425 with bad export cookie 7387247938117140389 [ 2350.667566] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2350.671798] LustreError: 93843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2351.059584] Lustre: server umount lustre-MDT0001 complete [ 2365.306449] Lustre: server umount lustre-OST0000 complete [ 2374.973757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.367895] LustreError: 98137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.519320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2379.704732] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2380.781855] LustreError: 98138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.898051] LustreError: 98137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.727239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.346041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.365701] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2398.473285] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2405.007956] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2416.339521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2416.485964] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2416.493548] LustreError: 98127:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2416.508263] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2416.513227] LustreError: 100220:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2416.519304] LustreError: 100220:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2416.525069] LustreError: 100220:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2416.564084] Lustre: server umount lustre-OST0000 complete [ 2416.566414] LustreError: 100220:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2424.125136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2424.343698] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2424.359335] LustreError: 98127:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2424.380989] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2424.402224] LustreError: 100682:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2424.416165] LustreError: 100682:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2424.427111] LustreError: 100682:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2424.516269] Lustre: server umount lustre-OST0000 complete [ 2424.520571] LustreError: 100682:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2433.224545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2433.389717] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2433.397160] Lustre: Found index 0 for lustre-OST0000, updating log [ 2433.409759] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2433.489870] Lustre: lustre-OST0000: new disk, initializing [ 2433.494163] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2433.502569] Lustre: 101216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2433.787453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2439.366008] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2442.498175] Lustre: Failing over lustre-OST0000 [ 2442.647732] Lustre: server umount lustre-OST0000 complete [ 2448.355426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2448.366739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.377657] Lustre: Skipped 1 previous similar message [ 2448.383087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.918611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.920130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.939267] Lustre: Skipped 1 previous similar message [ 2451.964833] Lustre: server umount lustre-MDT0000 complete [ 2454.972159] LustreError: 98118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389529 with bad export cookie 7387247938117143133 [ 2454.981402] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2455.236425] Lustre: server umount lustre-MDT0001 complete [ 2463.262593] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:05:36 (1787389536) [ 2477.577733] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2486.691689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.054624] LustreError: 103558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.225160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.268038] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2498.735926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.434901] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2506.028628] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2511.873826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.033522] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2525.994504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2526.131467] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2526.214436] Lustre: lustre-OST0001: new disk, initializing [ 2526.217708] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2526.224364] Lustre: 105873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2526.490605] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2526.497839] Lustre: Skipped 2 previous similar messages [ 2532.572153] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2533.549209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2533.574055] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2533.747993] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2539.413058] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2545.055215] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2549.221351] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.227706] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2549.232028] Lustre: Skipped 2 previous similar messages [ 2549.243550] Lustre: Skipped 2 previous similar messages [ 2554.338355] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2554.356537] Lustre: Skipped 3 previous similar messages [ 2555.345240] Lustre: server umount lustre-MDT0001 complete [ 2555.542513] LustreError: 103554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.561882] LustreError: 103554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2563.929390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2564.016249] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2564.021109] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2564.067188] Lustre: 103546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2565.474578] LustreError: 107420:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2565.489777] LustreError: 107420:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2565.497667] LustreError: 103551:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2565.642710] LustreError: 107430:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.112@tcp: cfg command failed: rc = -17 [ 2565.662120] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2565.662120] [ 2569.532357] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2570.222565] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2577.531956] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:07:31 (1787389651) [ 2592.323060] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2607.910941] Lustre: 109488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2626.583917] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2629.713588] Lustre: 110622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2707.937132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2707.948604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2707.975312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2710.063410] Lustre: server umount lustre-MDT0000 complete [ 2713.059876] LustreError: 103540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389787 with bad export cookie 7387247938117144057 [ 2713.063440] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2713.071718] LustreError: 103540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2713.319154] Lustre: server umount lustre-MDT0001 complete [ 2725.519309] Lustre: server umount lustre-OST0000 complete [ 2739.092329] Lustre: server umount lustre-OST0001 complete [ 2744.254243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2752.239843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2760.052787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2768.471741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2786.505069] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2794.871029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.898361] Lustre: lustre-MDT0000: reset Object Index mappings [ 2795.296764] LustreError: 114574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2795.305985] LustreError: 114574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2795.407331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.412348] Lustre: Skipped 1 previous similar message [ 2799.137271] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2806.105104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.132537] Lustre: lustre-MDT0001: reset Object Index mappings [ 2810.057284] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2812.360831] Lustre: 115720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2818.209368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2820.845456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2823.802610] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2830.808742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2832.253838] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2832.257547] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2832.259425] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2836.506368] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2843.091298] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2846.346516] Lustre: 117598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2889.910401] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:12:43 (1787389963) [ 2918.881238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2918.888089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.898418] Lustre: Skipped 3 previous similar messages [ 2918.903875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2918.909436] Lustre: Skipped 4 previous similar messages [ 2922.934647] Lustre: server umount lustre-MDT0000 complete [ 2924.521305] LustreError: 117619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2924.540399] LustreError: 117619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2925.872313] LustreError: 117600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390000 with bad export cookie 7387247938117190908 [ 2925.884147] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2925.886298] LustreError: 117600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2940.384459] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2940.713151] Lustre: server umount lustre-MDT0001 complete [ 2958.304242] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2958.428722] Lustre: server umount lustre-OST0000 complete [ 2976.227313] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2976.409984] Lustre: server umount lustre-OST0001 complete [ 2983.518080] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 2993.117640] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3029.889057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.130773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3030.160298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3030.250465] Lustre: lustre-MDT0000: new disk, initializing [ 3030.358120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3030.363236] Lustre: Skipped 3 previous similar messages [ 3030.377182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3034.842801] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3042.398549] Lustre: server umount lustre-MDT0000 complete [ 3052.425249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.728748] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3052.765150] Lustre: 121875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3052.789481] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3057.850752] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3062.974561] Lustre: server umount lustre-MDT0000 complete [ 3070.896247] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:15:44 (1787390144) [ 3072.586462] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3074.250817] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:15:47 (1787390147) [ 3091.151538] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3099.781966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.043028] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3112.255379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.347308] Lustre: 123923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3112.401784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3112.407597] Lustre: Skipped 1 previous similar message [ 3112.505907] Lustre: lustre-MDT0001: new disk, initializing [ 3112.697607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3112.705110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3117.343807] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3122.104140] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3128.430583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3128.680201] Lustre: lustre-OST0000: new disk, initializing [ 3128.684707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3128.692887] Lustre: 125501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3130.384824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3130.395285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3130.496116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3135.128974] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3145.919039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3146.111411] Lustre: lustre-OST0001: new disk, initializing [ 3146.114865] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3146.120350] Lustre: 126372:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3148.093796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3148.109793] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3148.173742] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3152.212704] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3161.650284] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3164.923681] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3530.331369] Lustre: 132778:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3530.338098] Lustre: 132778:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3530.342375] Lustre: 132778:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3530.346651] Lustre: 132778:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3530.351322] Lustre: 132778:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3530.355118] Lustre: 132778:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3785.458842] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:27:38 (1787390858) [ 3803.629964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.648179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3803.657547] Lustre: Skipped 6 previous similar messages [ 3803.666354] Lustre: Skipped 10 previous similar messages [ 3808.744634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3808.752434] Lustre: Skipped 6 previous similar messages [ 3814.368315] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3814.645987] Lustre: server umount lustre-MDT0000 complete [ 3817.599562] LustreError: 123916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390892 with bad export cookie 7387247938117257338 [ 3817.600653] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3817.608718] LustreError: 123916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3818.977763] LustreError: 127208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3818.981737] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.987680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3818.987688] Lustre: Skipped 4 previous similar messages [ 3818.995265] LustreError: 127208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3819.025584] Lustre: Skipped 3 previous similar messages [ 3824.025072] Lustre: server umount lustre-MDT0001 complete [ 3833.196202] Lustre: server umount lustre-OST0000 complete [ 3842.870246] Lustre: server umount lustre-OST0001 complete [ 3848.859842] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 3857.502560] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3892.509778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.757785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3892.793429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3892.865558] Lustre: lustre-MDT0000: new disk, initializing [ 3892.931486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.936880] Lustre: Skipped 5 previous similar messages [ 3892.957783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3897.302498] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3906.936593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.130548] Lustre: 135827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3907.229477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3907.238635] Lustre: Skipped 1 previous similar message [ 3907.356956] Lustre: lustre-MDT0001: new disk, initializing [ 3907.473597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3907.480675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3911.248801] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3919.019634] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3924.729462] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3931.935442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3932.294374] Lustre: lustre-OST0000: new disk, initializing [ 3932.297547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3932.308466] Lustre: 137795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3933.591633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3933.606470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3933.697589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3938.289145] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3946.733121] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3956.488693] Lustre: lustre-MDD0000: changelog on [ 3959.780298] Lustre: lustre-MDD0001: changelog on [ 5592.082786] Lustre: lustre-MDD0001: changelog off [ 5593.948824] Lustre: 135833:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5594.975502] Lustre: lustre-MDD0000: changelog off [ 5602.789837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5602.802355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5602.817325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5602.822653] Lustre: Skipped 4 previous similar messages [ 5605.860923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5605.867161] Lustre: Skipped 1 previous similar message [ 5607.145619] Lustre: server umount lustre-OST0000 complete [ 5610.978482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.982637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5610.991849] Lustre: Skipped 2 previous similar messages [ 5611.003448] Lustre: Skipped 1 previous similar message [ 5616.922430] Lustre: server umount lustre-MDT0000 complete [ 5620.524996] LustreError: 135819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392695 with bad export cookie 7387247938118077703 [ 5620.527288] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5620.536738] LustreError: 135819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5620.929667] Lustre: server umount lustre-MDT0001 complete [ 5631.664826] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 5634.488861] Key type lgssc unregistered [ 5634.912570] LNet: 146895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5634.924341] LNetError: 146895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5634.940995] LNet: Removed LNI 192.168.206.112@tcp [ 5635.826444] Key type .llcrypt unregistered [ 5635.829973] Key type ._llcrypt unregistered [ 5656.767839] Key type ._llcrypt registered [ 5656.769764] Key type .llcrypt registered [ 5656.895618] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 5671.912493] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5673.043279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5673.098146] alg: No test for adler32 (adler32-zlib) [ 5674.194404] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 5674.418734] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 5676.080160] Key type lgssc registered [ 5677.405042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5707.892391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5707.947342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.155159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5709.175687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5709.241437] Lustre: lustre-MDT0000: new disk, initializing [ 5709.296092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5709.311802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5713.141915] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5722.897828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.987561] Lustre: 150097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5723.033898] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5723.037720] Lustre: Skipped 1 previous similar message [ 5723.106979] Lustre: lustre-MDT0001: new disk, initializing [ 5723.231114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5723.287976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5723.319264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5727.471578] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5735.992685] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5741.203656] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5747.072652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5747.301975] Lustre: lustre-OST0000: new disk, initializing [ 5747.305534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5747.310248] Lustre: 152066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5747.398560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5752.817845] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5754.963546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5754.970108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5755.101496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5761.124816] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5766.555833] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_module obdecho/obdecho [ 5768.448467] LustreError: 153070:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5768.454918] LustreError: 153070:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5770.234986] LustreError: 153166:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5784.545432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5784.563443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.598424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5786.107950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5786.115598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5786.127923] Lustre: Skipped 1 previous similar message [ 5786.139104] Lustre: Skipped 2 previous similar messages [ 5787.957352] Lustre: server umount lustre-MDT0000 complete [ 5791.944341] LustreError: 150088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392866 with bad export cookie 9547277179331903199 [ 5791.946040] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5791.952599] LustreError: 150088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5792.362513] Lustre: server umount lustre-MDT0001 complete [ 5806.609578] Lustre: server umount lustre-OST0000 complete [ 5829.502086] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 5839.768825] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5875.127935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5875.336820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5875.366211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5875.427917] Lustre: lustre-MDT0000: new disk, initializing [ 5875.486609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5875.503215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5879.640823] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5890.531708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5890.660432] Lustre: 156898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5890.707053] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5890.710887] Lustre: Skipped 1 previous similar message [ 5890.772964] Lustre: lustre-MDT0001: new disk, initializing [ 5890.848517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5890.891366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5890.912233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5895.231892] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5905.330344] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5910.956597] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5917.588919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5917.995737] Lustre: lustre-OST0000: new disk, initializing [ 5917.999925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5918.010418] Lustre: 158866:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5918.108151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.314753] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5925.949226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5925.957399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5926.097848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5933.316525] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5937.651425] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5937.871899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5942.153528] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5942.398651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5946.852106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.857393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5946.858184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5946.876523] Lustre: Skipped 1 previous similar message [ 5950.263566] Lustre: server umount lustre-OST0000 complete [ 5957.099808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5957.111622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5957.120701] Lustre: Skipped 2 previous similar messages [ 5959.952621] Lustre: server umount lustre-MDT0000 complete [ 5962.209800] LustreError: 156909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.233362] LustreError: 156909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5963.301926] LustreError: 156888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393038 with bad export cookie 9547277179331904158 [ 5963.308722] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5963.313702] LustreError: 156888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5963.681090] Lustre: server umount lustre-MDT0001 complete [ 5967.686985] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:04:00 (1787393040) [ 5969.583930] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5971.880571] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:04:04 (1787393044) [ 5980.116700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5980.697781] LustreError: 161278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5980.775654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5985.763415] LustreError: 161279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5986.502689] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5990.897444] LustreError: 161278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5995.779326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5996.004249] LustreError: 161279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.796625] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6007.906189] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6013.327656] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6019.781993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6020.197826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6020.203160] Lustre: Skipped 1 previous similar message [ 6026.199208] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6033.612239] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6039.729948] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 6041.078116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.098269] Lustre: Skipped 2 previous similar messages [ 6043.491421] Lustre: server umount lustre-MDT0000 complete [ 6044.833607] LustreError: 161275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.817767] LustreError: 161258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393122 with bad export cookie 9547277179331905033 [ 6047.826040] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6047.829958] LustreError: 161258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6049.975267] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.12@tcp (stopping) [ 6049.983839] Lustre: Skipped 3 previous similar messages [ 6051.297685] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6051.319203] Lustre: Skipped 1 previous similar message [ 6054.189509] Lustre: server umount lustre-MDT0001 complete [ 6060.751686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6074.813905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6075.069508] LustreError: 165119:0:(llog.c:1655:llog_backup()) MGC192.168.206.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6075.076138] Lustre: 165119:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6077.921857] LustreError: 165124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6077.942835] LustreError: 165124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6110.362461] LustreError: 165125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.381587] LustreError: 165125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 6135.776711] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6135.783504] Lustre: 165119:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.112@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6175.895313] LustreError: 165124:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6175.915725] LustreError: 165124:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 6176.749180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6180.406651] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6181.872026] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6187.994951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6193.129732] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x847ebe02ce459f46 [ 6193.162466] Lustre: MGC192.168.206.112@tcp: Connection restored to 0@lo (at 0@lo) [ 6197.421203] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6198.777537] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6205.663614] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6207.677413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6213.585657] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6215.000414] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6234.594471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6234.601333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6234.610973] LustreError: Skipped 1 previous similar message [ 6234.613471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6234.619514] Lustre: Skipped 1 previous similar message [ 6234.628777] Lustre: Skipped 4 previous similar messages [ 6247.392126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6247.786470] Lustre: server umount lustre-MDT0000 complete [ 6252.201460] LustreError: 164061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393327 with bad export cookie 9547277179331911494 [ 6252.212314] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6252.222245] LustreError: 164061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6252.739960] Lustre: server umount lustre-MDT0001 complete [ 6268.272707] Lustre: server umount lustre-OST0000 complete [ 6285.955407] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 6296.546958] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 6339.762107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.179766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6340.225294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6340.337942] Lustre: lustre-MDT0000: new disk, initializing [ 6340.409165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6340.413520] Lustre: Skipped 1 previous similar message [ 6340.424911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6345.379530] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6355.762846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6355.841410] Lustre: 170230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6356.059733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6359.567861] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6370.248566] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6376.216121] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6382.994810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6383.255195] Lustre: lustre-OST0000: new disk, initializing [ 6383.258281] Lustre: Skipped 1 previous similar message [ 6383.261506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6383.265303] Lustre: Skipped 2 previous similar messages [ 6383.271609] Lustre: 172199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6389.445489] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6391.903255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6391.912590] Lustre: Skipped 1 previous similar message [ 6391.917573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6392.084274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6397.792220] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6402.190480] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6402.411081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6406.545826] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6406.742648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6412.769193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6412.773903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.777830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6412.784692] Lustre: Skipped 2 previous similar messages [ 6412.787364] Lustre: Skipped 6 previous similar messages [ 6414.582707] Lustre: server umount lustre-OST0000 complete [ 6422.498031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6422.508432] LustreError: Skipped 1 previous similar message [ 6424.720935] Lustre: server umount lustre-MDT0000 complete [ 6428.133409] LustreError: 170236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6428.172866] LustreError: 170236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [ 6428.673908] LustreError: 170220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393503 with bad export cookie 9547277179331928322 [ 6428.681441] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6428.687879] LustreError: 170220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6429.209143] Lustre: server umount lustre-MDT0001 complete [ 6437.429932] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:11:51 (1787393511) [ 6443.410530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6447.725785] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6455.598501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6459.952716] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6466.649472] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6471.920638] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6478.657108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6485.354813] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6493.036938] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6503.117515] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:12:56 (1787393576) [ 6509.025815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.032466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6509.039370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6509.041743] Lustre: Skipped 3 previous similar messages [ 6509.063716] Lustre: Skipped 5 previous similar messages [ 6513.487495] Lustre: server umount lustre-OST0000 complete [ 6523.567664] Lustre: server umount lustre-MDT0000 complete [ 6528.541889] LustreError: 174591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393603 with bad export cookie 9547277179331929204 [ 6528.550766] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6528.553814] LustreError: 174591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6529.066452] Lustre: server umount lustre-MDT0001 complete [ 6542.618309] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 6545.468537] Key type lgssc unregistered [ 6545.797785] LNet: 178341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6545.806465] LNetError: 178341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6545.825387] LNet: Removed LNI 192.168.206.112@tcp [ 6546.790520] Key type .llcrypt unregistered [ 6546.793095] Key type ._llcrypt unregistered [ 6565.362983] Key type ._llcrypt registered [ 6565.365425] Key type .llcrypt registered [ 6565.445950] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 6566.606923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6566.633848] alg: No test for adler32 (adler32-zlib) [ 6567.601739] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 6567.810092] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 6569.432144] Key type lgssc registered [ 6570.582780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6581.056202] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6581.074636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6596.642533] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6601.761485] LustreError: 179584:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.112@tcp: failed processing log, type 4: rc = -110 [ 6606.883235] LustreError: 179584:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6606.899745] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6606.907453] LustreError: MGC192.168.206.112@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6606.921842] LustreError: 179584:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6606.932190] LustreError: 179584:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6606.946896] LustreError: 179584:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6607.063711] Lustre: server umount lustre-OST0000 complete [ 6607.068731] LustreError: 179584:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6615.910720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.204739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6621.225156] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6626.790159] LustreError: 180084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6626.824310] LustreError: 180084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6630.941633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.689278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6638.082461] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6647.359318] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6653.526297] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6659.727272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6659.897832] LustreError: 181901:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6659.918106] Lustre: 181901:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.112@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6660.089412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6665.881380] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6672.920263] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6676.970144] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.992061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6676.993899] Lustre: Skipped 2 previous similar messages [ 6677.017767] Lustre: Skipped 2 previous similar messages [ 6682.088497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6682.094457] Lustre: Skipped 2 previous similar messages [ 6687.206492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6687.222199] Lustre: Skipped 2 previous similar messages [ 6689.248125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6689.548605] Lustre: server umount lustre-MDT0000 complete [ 6692.322422] LustreError: 180812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6692.333654] LustreError: 180812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6693.893853] LustreError: 181928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393768 with bad export cookie 12400651303741942595 [ 6693.894198] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6693.902723] LustreError: 181928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6695.520292] LustreError: 180814:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006724bba6 x1874218075584256/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6695.761158] Lustre: server umount lustre-MDT0001 complete [ 6710.302561] Lustre: server umount lustre-OST0000 complete [ 6719.497262] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 6730.501642] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 6767.909528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.230750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6768.287763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6768.412719] Lustre: lustre-MDT0000: new disk, initializing [ 6768.473382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6768.498668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6773.582290] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6783.304304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.419207] Lustre: 185427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6783.470170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6783.474919] Lustre: Skipped 1 previous similar message [ 6783.521579] Lustre: lustre-MDT0001: new disk, initializing [ 6783.591051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6783.627913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6783.640413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6787.718182] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6796.894947] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6802.846366] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6810.574280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6811.062118] Lustre: lustre-OST0000: new disk, initializing [ 6811.075946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6811.084263] Lustre: 187395:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6811.190039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6816.806582] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6820.390504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6820.397555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6820.474616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6825.634164] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6829.863439] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6830.125625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6834.988625] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6835.208589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6840.804959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6840.810424] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6840.821032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6840.843032] Lustre: Skipped 1 previous similar message [ 6843.434906] Lustre: server umount lustre-OST0000 complete [ 6850.019249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6850.029129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6850.041808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6850.045281] Lustre: Skipped 1 previous similar message [ 6853.861560] Lustre: server umount lustre-MDT0000 complete [ 6856.165038] LustreError: 186671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6856.189439] LustreError: 186671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6857.524612] LustreError: 185421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393932 with bad export cookie 12400651303741943316 [ 6857.532470] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6857.535684] LustreError: 185421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6857.868585] Lustre: server umount lustre-MDT0001 complete [ 6866.458541] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:18:59 (1787393939) [ 6884.269268] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 6886.838789] Key type lgssc unregistered [ 6887.179962] LNet: 190345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6887.185830] LNetError: 190345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6887.200691] LNet: Removed LNI 192.168.206.112@tcp [ 6888.179633] Key type .llcrypt unregistered [ 6888.186883] Key type ._llcrypt unregistered [ 6906.861464] Key type ._llcrypt registered [ 6906.863753] Key type .llcrypt registered [ 6906.983068] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 6908.228829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6908.275350] alg: No test for adler32 (adler32-zlib) [ 6909.354855] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 6909.677623] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 6911.360299] Key type lgssc registered [ 6912.594358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6921.290149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6921.322756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.717151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6927.556412] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6932.963828] LustreError: 191563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.987137] LustreError: 191563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6936.465093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.934526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6942.066969] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6949.941926] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6955.932536] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6963.175461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6963.338271] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6963.341265] LustreError: 193427:0:(llog.c:1696:llog_backup()) MGC192.168.206.112@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6963.349454] Lustre: 193427:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.112@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6963.490355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6969.966770] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6977.413693] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6981.799466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.812303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6981.828679] Lustre: Skipped 2 previous similar messages [ 6981.859628] Lustre: Skipped 2 previous similar messages [ 6985.965912] Lustre: server umount lustre-MDT0000 complete [ 6987.235227] LustreError: 191557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6987.257390] LustreError: 191557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6989.444814] LustreError: 191544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394064 with bad export cookie 14659375821679577756 [ 6989.458948] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6989.461924] LustreError: 191544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6990.014176] LustreError: 192293:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000221f1884 x1874218434194176/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6990.240318] Lustre: server umount lustre-MDT0001 complete [ 7004.231633] Lustre: server umount lustre-OST0000 complete [ 7012.448621] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 7021.171135] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 7056.119533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.258991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7056.281444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7056.352307] Lustre: lustre-MDT0000: new disk, initializing [ 7056.430289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7056.447484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7061.144425] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7071.484032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7071.658594] Lustre: 196951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7071.732995] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7071.742237] Lustre: Skipped 1 previous similar message [ 7071.755379] LustreError: 197733:0:(llog.c:1696:llog_backup()) MGC192.168.206.112@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7071.774815] LustreError: 197733:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7071.787723] Lustre: 197733:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.112@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7071.849856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7071.862387] Lustre: Skipped 1 previous similar message [ 7071.998567] Lustre: lustre-MDT0001: new disk, initializing [ 7072.229910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7072.297352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7072.331186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7076.348315] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7086.348895] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7092.733644] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7101.061928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7101.406306] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7101.410767] Lustre: Skipped 1 previous similar message [ 7101.415978] LustreError: 198895:0:(llog.c:1696:llog_backup()) MGC192.168.206.112@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7101.430438] LustreError: 198895:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7101.437548] Lustre: 198895:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.112@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7101.453901] Lustre: 198895:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7101.525587] Lustre: lustre-OST0000: new disk, initializing [ 7101.548093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7101.561121] Lustre: 198918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7101.781029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7108.213713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7108.237786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7108.396463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7108.943353] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7118.788362] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7122.927842] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7123.184980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7127.692477] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7127.957835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7134.180686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.195515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7134.238386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7134.251014] Lustre: Skipped 1 previous similar message [ 7136.377605] Lustre: server umount lustre-OST0000 complete [ 7143.908575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7143.918837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7143.950336] Lustre: Skipped 1 previous similar message [ 7143.962042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7148.624635] Lustre: server umount lustre-MDT0000 complete [ 7149.544279] LustreError: 196959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7149.557979] LustreError: 196959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7152.384241] LustreError: 196944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394227 with bad export cookie 14659375821679578484 [ 7152.387422] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7152.404377] LustreError: 196944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7152.837631] Lustre: server umount lustre-MDT0001 complete [ 7161.866811] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:23:55 (1787394235) [ 7178.194980] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 7187.793529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.176772] LustreError: 202046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.241961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7192.961818] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7193.570593] LustreError: 202047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.658660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.298748] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7208.981432] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7215.628280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7222.241981] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7231.512318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7231.791785] Lustre: lustre-OST0001: new disk, initializing [ 7231.795325] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7231.803710] Lustre: 204352:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7231.890585] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7231.894940] Lustre: Skipped 2 previous similar messages [ 7238.268547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7238.282390] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7238.390167] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7238.669590] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7249.682974] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7255.415311] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7257.210267] Lustre: DEBUG MARKER: ADD OST3 [ 7262.328560] loop: module loaded [ 7262.389456] loop0: detected capacity change from 0 to 204800000 [ 7262.642327] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7262.660649] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7262.671394] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7268.536410] loop0: detected capacity change from 0 to 204800000 [ 7269.302533] Lustre: DEBUG MARKER: STOP OST3 [ 7274.188683] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7274.482677] Lustre: lustre-OST0003: new disk, initializing [ 7274.488956] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7274.498445] Lustre: 206355:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7276.442408] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7276.604823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7276.618405] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7276.655089] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7278.103957] Lustre: Failing over lustre-OST0003 [ 7278.335965] Lustre: server umount lustre-OST0003 complete [ 7278.498099] LustreError: 206544:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7280.820524] LustreError: 203533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7280.837047] LustreError: 203533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7281.309775] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7281.636604] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7281.653860] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7281.673947] Lustre: Skipped 2 previous similar messages [ 7289.666323] Lustre: DEBUG MARKER: START OST3 again [ 7296.210300] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7296.739991] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7296.745182] Lustre: Skipped 1 previous similar message [ 7296.760272] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7297.840551] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7297.935198] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7297.943833] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7303.549970] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7324.808476] Lustre: Failing over lustre-OST0003 [ 7324.875443] Lustre: server umount lustre-OST0003 complete [ 7327.724071] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7327.730310] LustreError: 203533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7327.742871] Lustre: Skipped 1 previous similar message [ 7327.766616] LustreError: 203533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7343.093351] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.100901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7343.108420] Lustre: Skipped 2 previous similar messages [ 7343.120053] Lustre: Skipped 5 previous similar messages [ 7348.203137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7348.208753] Lustre: Skipped 3 previous similar messages [ 7356.386579] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7356.659036] Lustre: server umount lustre-MDT0000 complete [ 7360.270808] LustreError: 202028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394435 with bad export cookie 14659375821679579366 [ 7360.302242] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7363.556603] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7363.556604] LustreError: 202042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7363.568430] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7363.568438] Lustre: Skipped 4 previous similar messages [ 7363.578163] Lustre: Skipped 2 previous similar messages [ 7363.612263] LustreError: 202042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7366.825358] Lustre: server umount lustre-MDT0001 complete [ 7371.191424] Lustre: server umount lustre-OST0000 complete [ 7376.039037] Lustre: server umount lustre-OST0001 complete [ 7383.124819] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 7391.398807] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 7429.833536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7430.197637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7430.223742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7430.303846] Lustre: lustre-MDT0000: new disk, initializing [ 7430.372118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7430.392668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7436.021426] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7446.579655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7446.689234] Lustre: 211811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7446.766899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7446.771803] Lustre: Skipped 1 previous similar message [ 7446.881250] Lustre: lustre-MDT0001: new disk, initializing [ 7447.105106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7447.125737] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7452.379602] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7462.094467] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7468.232344] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7475.434193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7475.891174] Lustre: 213780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7477.297443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7477.491540] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7482.588876] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7495.085553] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7500.794603] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7501.055231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7506.184427] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7506.475980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7513.571331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7513.572899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.579694] LustreError: Skipped 1 previous similar message [ 7513.591610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7513.594354] Lustre: Skipped 1 previous similar message [ 7513.625389] Lustre: Skipped 4 previous similar messages [ 7523.297382] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7523.447457] Lustre: server umount lustre-OST0000 complete [ 7528.931776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7533.769205] Lustre: server umount lustre-MDT0000 complete [ 7534.061962] LustreError: 211823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7534.082255] LustreError: 211823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7537.277475] LustreError: 212606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394612 with bad export cookie 14659375821679581662 [ 7537.281178] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7537.287639] LustreError: 212606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7537.669675] Lustre: server umount lustre-MDT0001 complete [ 7541.177095] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:30:14 (1787394614) [ 7556.074204] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 7566.539768] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 7600.746819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7600.976765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7600.997610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7601.004143] Lustre: Skipped 1 previous similar message [ 7601.069503] Lustre: lustre-MDT0000: new disk, initializing [ 7601.072541] Lustre: Skipped 1 previous similar message [ 7601.158632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7601.165243] Lustre: Skipped 1 previous similar message [ 7605.884781] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7615.289141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.427096] Lustre: 218390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7615.697135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7620.118906] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7629.039479] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7634.613814] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7640.388955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7640.643672] Lustre: 220361:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7642.282575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7646.905730] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7657.002713] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7661.709083] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7662.143629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7667.075448] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7667.218958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7673.314736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7673.317220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7673.323600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7673.323608] Lustre: Skipped 4 previous similar messages [ 7673.336073] Lustre: Skipped 3 previous similar messages [ 7683.043129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7683.335865] Lustre: server umount lustre-OST0000 complete [ 7693.364202] Lustre: server umount lustre-MDT0000 complete [ 7693.793680] LustreError: 218403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7693.832672] LustreError: 218403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7696.666580] LustreError: 218383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394771 with bad export cookie 14659375821679582544 [ 7696.669396] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7696.679967] LustreError: 218383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7696.954306] Lustre: server umount lustre-MDT0001 complete [ 7704.045791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7704.416171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7704.421467] Lustre: Skipped 5 previous similar messages [ 7708.122393] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7715.191285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7719.972218] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7727.669812] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7733.150706] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7738.952880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7745.682497] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7754.285527] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7764.964770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7764.980217] Lustre: Skipped 6 previous similar messages [ 7770.665285] Lustre: server umount lustre-OST0000 complete [ 7781.291232] Lustre: server umount lustre-MDT0000 complete [ 7784.460446] LustreError: 222559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394859 with bad export cookie 14659375821679583426 [ 7784.461626] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7784.466943] LustreError: 222559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7784.687525] Lustre: server umount lustre-MDT0001 complete [ 7892.642242] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 7895.708588] Key type lgssc unregistered [ 7896.040584] LNet: 226245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7896.047409] LNetError: 226245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7896.063372] LNet: Removed LNI 192.168.206.112@tcp [ 7896.889157] Key type .llcrypt unregistered [ 7896.891932] Key type ._llcrypt unregistered [ 7921.591821] Key type ._llcrypt registered [ 7921.594115] Key type .llcrypt registered [ 7921.707211] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 7936.989521] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 7938.094878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7938.336431] alg: No test for adler32 (adler32-zlib) [ 7939.453586] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 7939.704919] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 7941.384162] Key type lgssc registered [ 7942.395301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7975.875499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7975.908648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7977.171634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7977.215892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7977.325590] Lustre: lustre-MDT0000: new disk, initializing [ 7977.402530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7977.428880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7981.735733] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7992.225479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7992.333382] Lustre: 229602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7992.401765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7992.410692] Lustre: Skipped 1 previous similar message [ 7992.497809] Lustre: lustre-MDT0001: new disk, initializing [ 7992.599076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7992.651141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7992.669751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7997.057745] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8005.955293] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8011.541161] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8017.248101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8017.469496] Lustre: lustre-OST0000: new disk, initializing [ 8017.473379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8017.485959] Lustre: 231569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8017.573823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8023.367721] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8027.228304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8027.237397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8027.335299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8032.653195] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8037.090510] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8037.285771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8042.048365] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8042.250863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8048.101918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8048.102102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.111024] LustreError: Skipped 1 previous similar message [ 8048.112725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8048.139092] Lustre: Skipped 1 previous similar message [ 8053.222432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8053.232774] Lustre: Skipped 2 previous similar messages [ 8058.336301] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8058.342398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8058.370257] Lustre: Skipped 1 previous similar message [ 8058.577702] Lustre: server umount lustre-OST0000 complete [ 8063.476335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8063.485263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8063.498251] Lustre: Skipped 1 previous similar message [ 8063.520439] Lustre: Skipped 1 previous similar message [ 8068.579453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8068.597712] Lustre: Skipped 1 previous similar message [ 8068.916117] Lustre: server umount lustre-MDT0000 complete [ 8072.801118] LustreError: 229595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395147 with bad export cookie 14707675688046024172 [ 8072.803330] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8072.812521] LustreError: 229595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8073.299702] Lustre: server umount lustre-MDT0001 complete [ 8094.219246] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 8105.569145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8106.196411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8111.388421] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8111.588580] LustreError: 234505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.621882] LustreError: 234505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8115.688169] LustreError: 234506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8120.719913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8120.805888] LustreError: 234505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8121.162650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8126.275316] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8129.201447] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8136.591725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8136.995179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8144.184176] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8153.985411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8154.184181] Lustre: lustre-OST0001: new disk, initializing [ 8154.194122] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8154.205264] Lustre: 236814:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8154.340143] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8160.872322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8160.882758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8160.961193] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8161.600262] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8174.542578] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8183.739900] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8196.781497] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 06:41:10 (1787395270) [ 8200.167695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8200.174555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8200.199108] Lustre: Skipped 3 previous similar messages [ 8200.230748] Lustre: Skipped 3 previous similar messages [ 8205.193941] Lustre: server umount lustre-MDT0000 complete [ 8205.294950] LustreError: 234502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8205.308365] LustreError: 234502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8209.081152] LustreError: 237581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395283 with bad export cookie 14707675688046025054 [ 8209.094787] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8209.097599] LustreError: 237581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8209.558381] Lustre: server umount lustre-MDT0001 complete [ 8224.057537] Lustre: server umount lustre-OST0000 complete [ 8226.272299] Lustre: 227828:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395285/real 1787395285] req@ffff96793c335f80 x1874219514314880/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787395301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8226.306428] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8228.014822] Lustre: server umount lustre-OST0001 complete [ 8235.214495] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 8247.156738] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 8283.896396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.137135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8284.159322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8284.270704] Lustre: lustre-MDT0000: new disk, initializing [ 8284.385207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.401738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8289.291973] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8300.966106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8301.121994] Lustre: 240759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8301.176550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8301.180665] Lustre: Skipped 1 previous similar message [ 8301.244825] Lustre: lustre-MDT0001: new disk, initializing [ 8301.443376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8301.458439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8306.371756] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8316.513847] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8322.603789] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8330.173158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8330.509889] Lustre: lustre-OST0000: new disk, initializing [ 8330.515161] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8330.520976] Lustre: 242728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8331.744649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8331.751783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8331.880505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8337.257167] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8347.037768] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8352.063127] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8352.293984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8356.906375] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8357.148604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8362.475614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8362.477039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8362.495457] LustreError: Skipped 1 previous similar message [ 8362.529390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8365.574579] Lustre: server umount lustre-OST0000 complete [ 8372.707809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8372.722286] Lustre: Skipped 1 previous similar message [ 8376.334404] Lustre: server umount lustre-MDT0000 complete [ 8377.832637] LustreError: 241637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8377.852919] LustreError: 241637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8380.280233] LustreError: 240753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395455 with bad export cookie 14707675688046027189 [ 8380.287535] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8380.310858] LustreError: 240753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8380.763823] Lustre: server umount lustre-MDT0001 complete [ 8389.326715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8389.831535] LustreError: 244944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8389.963394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8389.981355] Lustre: Skipped 2 previous similar messages [ 8395.887350] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8405.374145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8410.789300] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8417.742876] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8423.000896] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8429.813498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8437.158808] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8444.991459] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8592.354907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.379096] Lustre: Skipped 1 previous similar message [ 8592.382660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8592.389802] Lustre: Skipped 4 previous similar messages [ 8598.514898] Lustre: server umount lustre-OST0000 complete [ 8608.762309] Lustre: server umount lustre-MDT0000 complete [ 8612.730820] LustreError: 244924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395687 with bad export cookie 14707675688046028071 [ 8612.739678] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8612.751510] LustreError: 244924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8613.089896] Lustre: server umount lustre-MDT0001 complete [ 8713.282177] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 8716.228228] Key type lgssc unregistered [ 8716.616995] LNet: 248561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8716.629773] LNetError: 248561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8716.665116] LNet: Removed LNI 192.168.206.112@tcp [ 8717.626315] Key type .llcrypt unregistered [ 8717.627870] Key type ._llcrypt unregistered [ 8744.477718] Key type ._llcrypt registered [ 8744.479406] Key type .llcrypt registered [ 8744.593407] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 8761.348714] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 8763.244416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8763.269066] alg: No test for adler32 (adler32-zlib) [ 8764.302880] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 8764.528993] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 8766.216162] Key type lgssc registered [ 8767.620434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8810.756449] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 8818.941693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8819.036985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.287811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8820.320479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8820.440294] Lustre: lustre-MDT0000: new disk, initializing [ 8820.551915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8820.592676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8826.159991] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8836.419994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8836.614333] Lustre: 252645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8836.675315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8836.680428] Lustre: Skipped 1 previous similar message [ 8836.807094] Lustre: lustre-MDT0001: new disk, initializing [ 8836.962430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8837.010459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8837.024477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8841.942781] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8847.065692] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8853.979937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8854.171061] Lustre: lustre-OST0000: new disk, initializing [ 8854.179610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8854.186868] Lustre: 254279:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8854.336120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8856.243833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8856.250586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8856.420753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8860.023932] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8870.957799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8871.268208] Lustre: lustre-OST0001: new disk, initializing [ 8871.276084] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8871.296921] Lustre: 255151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8871.537080] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8876.600247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8876.606069] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8876.688887] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8878.273151] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8890.941201] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8897.435335] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8908.772351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8908.788274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8908.826145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8912.871398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8912.873565] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8912.879275] Lustre: Skipped 1 previous similar message [ 8912.905522] Lustre: Skipped 2 previous similar messages [ 8914.949107] Lustre: server umount lustre-MDT0000 complete [ 8918.803795] LustreError: 252640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395993 with bad export cookie 14076585509059643394 [ 8918.818304] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8918.833908] LustreError: 252640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8919.226418] Lustre: server umount lustre-MDT0001 complete [ 8932.467200] Lustre: server umount lustre-OST0000 complete [ 8946.276636] Lustre: server umount lustre-OST0001 complete [ 8959.343496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8968.749707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8976.506281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8985.895164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9005.804397] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9016.621933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.650289] Lustre: lustre-MDT0000: reset Object Index mappings [ 9017.246044] LustreError: 260160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9017.269409] LustreError: 260160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9017.345208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9022.405651] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9022.437123] LustreError: 260161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9027.555306] LustreError: 260160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9031.412693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9031.427863] Lustre: lustre-MDT0001: reset Object Index mappings [ 9031.766446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9036.596654] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9039.867537] Lustre: 261302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9046.727112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9053.258370] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9056.245206] LustreError: 261654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.346858] LustreError: 261657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.372044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9061.699599] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9061.709300] Lustre: Skipped 1 previous similar message [ 9062.721296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9062.726714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9068.380962] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9076.041297] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9079.421621] Lustre: 263174:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9086.614152] Lustre: server umount lustre-MDT0000 complete [ 9087.973278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9087.984487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9088.008202] LustreError: 260160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9090.169440] LustreError: 260140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396164 with bad export cookie 14076585509059650184 [ 9090.170267] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9090.177847] LustreError: 260140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9090.693263] Lustre: server umount lustre-MDT0001 complete [ 9104.494939] Lustre: server umount lustre-OST0000 complete [ 9117.511602] Lustre: server umount lustre-OST0001 complete [ 9136.212354] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 06:56:49 (1787396209) [ 9152.043927] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 9162.875550] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9199.450452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9199.685564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9199.718561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9199.826286] Lustre: lustre-MDT0000: new disk, initializing [ 9199.916856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9199.939791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9204.757908] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9216.046165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9216.196578] Lustre: 267427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9216.277288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9216.283563] Lustre: Skipped 1 previous similar message [ 9216.379946] Lustre: lustre-MDT0001: new disk, initializing [ 9216.594640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9216.611169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9221.327120] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9230.988509] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9237.157610] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9243.816164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9244.102817] Lustre: lustre-OST0000: new disk, initializing [ 9244.109175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9244.113936] Lustre: 269397:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9245.859669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9245.869528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9246.024836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9250.592720] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9260.554159] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9265.638761] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9265.954206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9270.756603] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9270.966434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9276.901897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9276.904630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9276.908713] LustreError: Skipped 1 previous similar message [ 9276.909820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9276.909825] Lustre: Skipped 2 previous similar messages [ 9276.981442] Lustre: Skipped 4 previous similar messages [ 9282.022040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9282.031150] Lustre: Skipped 2 previous similar messages [ 9287.138164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9287.142147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9287.155776] Lustre: Skipped 1 previous similar message [ 9287.276836] Lustre: server umount lustre-OST0000 complete [ 9292.258850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9297.188332] Lustre: server umount lustre-MDT0000 complete [ 9297.380523] LustreError: 267439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9297.399648] LustreError: 267439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9300.908969] LustreError: 267418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396375 with bad export cookie 14076585509059652277 [ 9300.910324] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9300.923185] LustreError: 267418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9301.237358] Lustre: server umount lustre-MDT0001 complete [ 9319.501359] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9329.288506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9329.827016] LustreError: 272332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9329.908778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9329.913910] Lustre: Skipped 2 previous similar messages [ 9334.035924] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9342.960408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9349.363206] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9352.702992] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9359.188253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9365.936146] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9374.251556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9374.448616] Lustre: lustre-OST0001: new disk, initializing [ 9374.451234] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9374.472191] Lustre: 274639:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9376.478825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9376.489775] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9376.548762] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9381.376093] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9392.085957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9395.784429] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9401.003744] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9401.228212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9407.462485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9407.466793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9407.482012] Lustre: Skipped 4 previous similar messages [ 9407.495608] Lustre: Skipped 5 previous similar messages [ 9414.100924] Lustre: server umount lustre-MDT0000 complete [ 9417.513163] LustreError: 275824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396492 with bad export cookie 14076585509059653159 [ 9417.518329] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9417.525488] LustreError: 275824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9417.696859] LustreError: 272327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9417.707487] LustreError: 272327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9417.755197] Lustre: server umount lustre-MDT0001 complete [ 9423.170522] Lustre: server umount lustre-OST0000 complete [ 9426.966717] Lustre: server umount lustre-OST0001 complete [ 9443.453263] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9455.100763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.046369] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9470.939911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9471.415212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9471.419934] Lustre: Skipped 4 previous similar messages [ 9475.121828] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9477.704854] Lustre: 278849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9484.418023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9491.427468] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9499.607236] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9500.951369] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9501.039370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9506.471276] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9513.195447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9516.680376] Lustre: 280724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9521.306051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9521.496938] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9530.291745] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:03:23 (1787396603) [ 9547.252134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9547.266127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9547.272414] Lustre: Skipped 4 previous similar messages [ 9547.286757] Lustre: Skipped 7 previous similar messages [ 9551.269763] Lustre: server umount lustre-MDT0000 complete [ 9552.365731] LustreError: 277709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9552.388703] LustreError: 277709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9555.047728] LustreError: 280030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396629 with bad export cookie 14076585509059659711 [ 9555.054761] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9555.060556] LustreError: 280030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9561.507908] Lustre: server umount lustre-MDT0001 complete [ 9571.635077] Lustre: server umount lustre-OST0000 complete [ 9581.902345] Lustre: server umount lustre-OST0001 complete [ 9589.209331] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 9599.608109] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9646.126219] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9656.219854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9656.509482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9656.541432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9656.633863] Lustre: lustre-MDT0000: new disk, initializing [ 9656.819578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9661.278483] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9671.457040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9671.652493] Lustre: 284966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9671.966279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9676.900977] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9687.953279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9688.272075] Lustre: 286600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9689.686806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9693.774057] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9704.816521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9704.981567] Lustre: 287472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9706.486936] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9706.498389] Lustre: Skipped 1 previous similar message [ 9706.650309] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9711.338487] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9720.093345] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9723.995343] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9723.998072] Lustre: Skipped 1 previous similar message [ 9737.312304] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:06:50 (1787396810) [ 9741.754314] Lustre: server umount lustre-MDT0000 complete [ 9742.306123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9742.324696] Lustre: Skipped 6 previous similar messages [ 9745.386814] LustreError: 284959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396820 with bad export cookie 14076585509059667383 [ 9745.392410] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9745.403816] LustreError: 284959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9745.826735] Lustre: server umount lustre-MDT0001 complete [ 9759.825374] Lustre: server umount lustre-OST0000 complete [ 9773.799571] Lustre: server umount lustre-OST0001 complete [ 9782.240991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9782.950348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9782.959966] Lustre: Skipped 6 previous similar messages [ 9787.624353] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9795.516768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9800.237577] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9807.485560] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9813.238658] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9819.434417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9820.848062] LustreError: 291546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9820.880070] LustreError: 291546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 9826.171843] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9833.407483] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9848.550185] loop1: detected capacity change from 0 to 204800000 [ 9848.755421] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9848.773196] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9848.789604] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9853.569845] loop1: detected capacity change from 0 to 204800000 [ 9858.875247] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.000678] LustreError: 289708:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs5589: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9859.009883] LustreError: 289708:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs5589 [ 9859.014501] LustreError: 289708:0:(mgs_handler.c:517:mgs_target_reg()) fs5589-MDT0001: Error checking target index: rc = -13 [ 9859.024522] LustreError: fs5589-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9859.030506] LustreError: 293435:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9859.034753] LustreError: 293435:0:(tgt_mount.c:2129:server_put_super()) no obd fs5589-MDT0001 [ 9859.041076] LustreError: 293435:0:(tgt_mount.c:116:server_deregister_mount()) fs5589-MDT0001 not registered [ 9859.066603] Lustre: server umount fs5589-MDT0001 complete [ 9859.069754] LustreError: 293435:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9867.675052] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9867.792690] Lustre: Setting parameter fs5589-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs5589-MDT0001 [ 9867.892268] Lustre: srv-fs5589-MDT0001: No data found on store. Initialize space. [ 9867.902792] Lustre: Skipped 4 previous similar messages [ 9868.002624] Lustre: fs5589-MDT0001: new disk, initializing [ 9868.010735] Lustre: Skipped 3 previous similar messages [ 9868.147192] Lustre: srv-fs5589-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9868.153042] Lustre: 293902:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5589-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9868.162972] LustreError: 293902:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5589-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9872.752181] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9879.152881] Lustre: server umount fs5589-MDT0001 complete [ 9892.164529] loop1: detected capacity change from 0 to 204800000 [ 9897.801119] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.376752] Lustre: 295469:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5589-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9898.384463] LustreError: 295469:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5589-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9902.835144] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9904.126823] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9907.734204] Lustre: server umount fs5589-MDT0001 complete [ 9909.244450] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9918.945432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9918.953808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9918.972422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9918.976515] Lustre: Skipped 4 previous similar messages [ 9923.654756] Lustre: server umount lustre-MDT0000 complete [ 9931.592539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9931.731922] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9936.501963] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9937.407567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9943.399480] loop1: detected capacity change from 0 to 204800000 [ 9949.226768] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9949.637394] Lustre: 297917:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5589-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9949.649618] LustreError: 297917:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5589-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9953.385699] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9957.891537] Lustre: server umount fs5589-MDT0001 complete [ 9979.876709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9983.468818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9983.475476] Lustre: Skipped 5 previous similar messages [ 9985.808645] Lustre: server umount lustre-OST0000 complete [ 9993.698981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9993.706642] LustreError: Skipped 1 previous similar message [ 9995.948557] Lustre: server umount lustre-MDT0000 complete [ 9999.504372] LustreError: 289699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397074 with bad export cookie 14076585509059670071 [ 9999.512927] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9999.516667] LustreError: 289699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9999.892950] Lustre: server umount lustre-MDT0001 complete [10012.249695] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [10016.252292] Key type lgssc unregistered [10016.578076] LNet: 300264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10016.584988] LNetError: 300264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10016.606819] LNet: Removed LNI 192.168.206.112@tcp [10017.505337] Key type .llcrypt unregistered [10017.510765] Key type ._llcrypt unregistered [10037.759827] Key type ._llcrypt registered [10037.765898] Key type .llcrypt registered [10037.870997] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [10038.868518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10038.959760] alg: No test for adler32 (adler32-zlib) [10039.990207] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [10040.179298] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [10041.856165] Key type lgssc registered [10043.044487] Lustre: Echo OBD driver; http://www.lustre.org/ [10050.021380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10050.049783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10051.486476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10055.276705] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10061.796728] LustreError: 301538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10061.812535] LustreError: 301538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10063.921762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10064.683682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10068.603976] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10076.829768] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10082.181885] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10088.319435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10088.885789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10089.899321] LustreError: 303368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10094.868559] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10095.097210] LustreError: 303368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10097.137300] LustreError: 303366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10097.167582] LustreError: 303366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10102.244774] LustreError: 303369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10102.275561] LustreError: 303369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10103.609495] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10110.619718] LustreError: 303368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10110.638871] LustreError: 303368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10122.622664] loop1: detected capacity change from 0 to 204800000 [10122.874864] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10122.888379] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10122.901503] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10127.855259] LustreError: 303447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10127.869577] LustreError: 303447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10128.455719] loop1: detected capacity change from 0 to 204800000 [10134.497332] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10134.669812] Lustre: fs30665-OST0000: new disk, initializing [10134.674580] Lustre: srv-fs30665-OST0000: No data found on store. Initialize space. [10134.685758] Lustre: 305273:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10134.770372] Lustre: fs30665-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10141.988617] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10150.709532] Lustre: server umount fs30665-OST0000 complete [10161.823392] LustreError: 303366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10161.853080] LustreError: 303366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [10162.450286] loop1: detected capacity change from 0 to 204800000 [10167.662719] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10167.784575] Lustre: fs30665-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10173.923526] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10175.173927] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10179.055112] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10184.988981] loop2: detected capacity change from 0 to 204800000 [10185.108700] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10185.114893] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10185.135822] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10191.346733] loop2: detected capacity change from 0 to 204800000 [10196.935068] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10197.019924] LustreError: fs30665-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10197.029370] LustreError: fs30665-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10197.036498] LustreError: 308316:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10197.041738] LustreError: 308316:0:(tgt_mount.c:2129:server_put_super()) no obd fs30665-OST0001 [10197.045204] LustreError: 308316:0:(tgt_mount.c:116:server_deregister_mount()) fs30665-OST0001 not registered [10197.080480] Lustre: server umount fs30665-OST0001 complete [10197.089770] LustreError: 308316:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10208.072416] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10208.303969] Lustre: fs30665-OST0001: new disk, initializing [10208.309545] Lustre: srv-fs30665-OST0001: No data found on store. Initialize space. [10208.325523] Lustre: 308868:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10208.465393] Lustre: fs30665-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10215.464875] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10224.460824] Lustre: server umount fs30665-OST0000 complete [10228.393089] LustreError: 303369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10228.421887] LustreError: 303369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10233.982853] Lustre: server umount fs30665-OST0001 complete [10246.610227] loop1: detected capacity change from 0 to 204800000 [10252.539457] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10252.845073] Lustre: fs30665-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10259.837860] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10267.427216] loop2: detected capacity change from 0 to 204800000 [10273.674259] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10280.981923] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10289.509267] Lustre: server umount fs30665-OST0000 complete [10298.299841] Lustre: server umount fs30665-OST0001 complete [10312.168451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10312.171310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10312.189197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10317.193322] Lustre: server umount lustre-OST0000 complete [10320.870023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10320.878076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.890582] Lustre: Skipped 1 previous similar message [10320.908729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10320.919503] Lustre: Skipped 1 previous similar message [10322.404222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10322.406213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10326.892104] Lustre: server umount lustre-MDT0000 complete [10330.544286] LustreError: 301520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397405 with bad export cookie 4671157113123091071 [10330.552820] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10330.557258] LustreError: 301520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10330.928035] Lustre: server umount lustre-MDT0001 complete [10342.802176] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [10345.799261] Key type lgssc unregistered [10346.233851] LNet: 314795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10346.241928] LNetError: 314795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10346.268251] LNet: Removed LNI 192.168.206.112@tcp [10347.239295] Key type .llcrypt unregistered [10347.242652] Key type ._llcrypt unregistered [10379.901666] Key type ._llcrypt registered [10379.904391] Key type .llcrypt registered [10380.086210] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [10396.012688] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [10397.349991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10397.506822] alg: No test for adler32 (adler32-zlib) [10398.838357] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [10399.095970] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [10400.794284] Key type lgssc registered [10402.152784] Lustre: Echo OBD driver; http://www.lustre.org/ [10433.678790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10433.710195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10434.941695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10434.981876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10435.062632] Lustre: lustre-MDT0000: new disk, initializing [10435.132167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10435.152252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10439.831397] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10450.251736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10450.396296] Lustre: 318661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10450.484471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10450.487869] Lustre: Skipped 1 previous similar message [10450.598336] Lustre: lustre-MDT0001: new disk, initializing [10450.767748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10450.826510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10450.846505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10455.796583] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10464.613748] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10470.184367] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10477.241594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10477.460991] Lustre: lustre-OST0000: new disk, initializing [10477.464462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10477.469804] Lustre: 320632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10477.563767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10483.027642] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10483.817197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10483.841348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10483.926320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10491.557796] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10493.267288] Lustre: Mounted lustre-client - version 2.17.57_45_g4630995 [10493.969214] Lustre: Unmounted lustre-client [10503.651222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10503.665176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10503.674833] Lustre: Skipped 1 previous similar message [10503.687216] Lustre: Skipped 1 previous similar message [10507.438724] Lustre: server umount lustre-OST0000 complete [10512.356583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10512.366278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10512.406056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10513.890472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.892722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10517.715238] Lustre: server umount lustre-MDT0000 complete [10521.277206] LustreError: 318655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397596 with bad export cookie 16054294606180012698 [10521.280821] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10521.288785] LustreError: 318655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10521.694667] Lustre: server umount lustre-MDT0001 complete [10531.779704] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [10534.493630] Key type lgssc unregistered [10555.808136] LNet: 1 local NIs in recovery (showing 1): 192.168.206.112@tcp [10556.261254] LNet: 322792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10556.269616] LNetError: 322792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10622.825458] LNet: Removed LNI 192.168.206.112@tcp [10623.613070] Key type .llcrypt unregistered [10623.617189] Key type ._llcrypt unregistered [10639.826397] Key type ._llcrypt registered [10639.831613] Key type .llcrypt registered [10639.948331] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [10652.305838] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [10653.061409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10653.081941] alg: No test for adler32 (adler32-zlib) [10654.078270] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [10654.341961] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [10656.056491] Key type lgssc registered [10657.224892] Lustre: Echo OBD driver; http://www.lustre.org/ [10688.103877] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:22:41 (1787397761) [10694.290514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10694.359887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10695.639081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10695.688483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10695.804370] Lustre: lustre-MDT0000: new disk, initializing [10695.906108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10695.924700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10701.043058] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10710.113560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10710.207386] Lustre: 325993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10710.260366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10710.265597] Lustre: Skipped 1 previous similar message [10710.334153] Lustre: lustre-MDT0001: new disk, initializing [10710.430813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10710.477463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10710.490911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10714.873578] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10723.198747] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10728.641407] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10735.016644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10735.419220] Lustre: lustre-OST0000: new disk, initializing [10735.422448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10735.427865] Lustre: 327965:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10735.517706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10742.322269] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10742.348820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10742.355962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10742.477612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10752.091815] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10768.357088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10768.364218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10768.366406] LustreError: Skipped 1 previous similar message [10768.372179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10768.401989] Lustre: Skipped 1 previous similar message [10771.911503] Lustre: server umount lustre-OST0000 complete [10777.057240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10777.067433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10777.092325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10777.095474] Lustre: Skipped 1 previous similar message [10778.595820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10778.596109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10781.662894] Lustre: server umount lustre-MDT0000 complete [10785.181688] LustreError: 325985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397859 with bad export cookie 17586934724226675142 [10785.193916] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10785.204047] LustreError: 325985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10785.712590] Lustre: server umount lustre-MDT0001 complete [10796.268250] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [10798.818529] Key type lgssc unregistered [10799.137238] LNet: 329875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10799.142395] LNetError: 329875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10799.167900] LNet: Removed LNI 192.168.206.112@tcp [10800.114406] Key type .llcrypt unregistered [10800.117240] Key type ._llcrypt unregistered [10821.614358] Key type ._llcrypt registered [10821.617209] Key type .llcrypt registered [10821.729858] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [10822.934159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10822.966502] alg: No test for adler32 (adler32-zlib) [10824.254355] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [10824.552682] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [10826.192161] Key type lgssc registered [10827.397119] Lustre: Echo OBD driver; http://www.lustre.org/ [10834.858730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10834.893962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10836.337315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10840.608412] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10846.689913] LustreError: 331150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.705954] LustreError: 331150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10849.296117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10849.735170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10854.043893] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10861.862931] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10868.175225] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10875.241732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10875.737812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10882.498872] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10889.591432] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10903.285505] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10904.932490] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10910.692535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10910.704360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10910.722888] Lustre: Skipped 1 previous similar message [10910.747941] Lustre: Skipped 2 previous similar messages [10912.934950] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.12@tcp (stopping) [10915.221549] Lustre: server umount lustre-MDT0000 complete [10915.815973] LustreError: 331877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10915.859336] LustreError: 331877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10918.040991] LustreError: 331146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.817365] LustreError: 331130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397993 with bad export cookie 3284166192234598800 [10918.826475] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10918.831658] LustreError: 331130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10920.932771] LustreError: 331144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10920.964748] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.983774] Lustre: Skipped 1 previous similar message [10920.995897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10923.202670] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.12@tcp (stopping) [10928.275682] LustreError: 331145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10928.298593] LustreError: 331145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10928.299234] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.12@tcp (stopping) [10928.329235] Lustre: Skipped 1 previous similar message [10933.216136] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10933.457558] Lustre: server umount lustre-MDT0001 complete [10937.096707] Lustre: server umount lustre-OST0000 complete [10944.986716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10949.514808] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10952.561677] Lustre: 335329:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.112@tcp [10954.614852] Lustre: server umount MGS complete [10962.227292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10962.732468] LustreError: 335900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10962.753326] LustreError: 335900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10962.825405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10966.970688] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10975.467296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10975.728188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10980.060156] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10988.187041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10988.499279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10994.558714] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11001.516073] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11002.945883] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11007.824436] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11011.398931] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [11016.161436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11016.172098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11016.197435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11016.199937] Lustre: Skipped 1 previous similar message [11018.209716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11018.226559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11021.893956] Lustre: server umount lustre-OST0000 complete [11026.913401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11026.925285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11032.363581] Lustre: server umount lustre-MDT0000 complete [11033.569758] LustreError: 335901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11033.595871] LustreError: 335901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11036.847316] LustreError: 335882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398111 with bad export cookie 3284166192234599640 [11036.851845] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11036.865191] LustreError: 335882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11037.097093] Lustre: server umount lustre-MDT0001 complete [11046.443065] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11049.581886] Key type lgssc unregistered [11049.845745] LNet: 339339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11049.862720] LNetError: 339339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11049.887677] LNet: Removed LNI 192.168.206.112@tcp [11050.936371] Key type .llcrypt unregistered [11050.940984] Key type ._llcrypt unregistered [11068.308401] Key type ._llcrypt registered [11068.310794] Key type .llcrypt registered [11068.443962] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [11084.737421] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [11085.421537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11085.461971] alg: No test for adler32 (adler32-zlib) [11086.466862] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [11086.712239] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [11088.368225] Key type lgssc registered [11089.273725] Lustre: Echo OBD driver; http://www.lustre.org/ [11119.717706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11119.745498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11120.972860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11121.011983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11121.106843] Lustre: lustre-MDT0000: new disk, initializing [11121.195438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11121.209355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11125.759521] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11135.995542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11136.115803] Lustre: 342500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11136.178643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11136.181981] Lustre: Skipped 1 previous similar message [11136.290778] Lustre: lustre-MDT0001: new disk, initializing [11136.367326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11136.404728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11136.420983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11140.280342] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11149.240492] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11154.449174] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11160.165327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11160.443769] Lustre: lustre-OST0000: new disk, initializing [11160.448281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11160.459450] Lustre: 344464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11160.519271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11166.184863] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11168.347797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11168.364497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11168.492406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11175.229878] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11179.722147] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11180.019579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11184.500186] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11184.679431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11189.220309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11189.229736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11189.246234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11189.256808] Lustre: Skipped 1 previous similar message [11192.628562] Lustre: server umount lustre-OST0000 complete [11197.921523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11197.933340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.944635] Lustre: Skipped 1 previous similar message [11197.955321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11199.457825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11199.458608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11202.441356] Lustre: server umount lustre-MDT0000 complete [11206.758377] LustreError: 342490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398281 with bad export cookie 10700128035069431697 [11206.788070] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11207.146601] Lustre: server umount lustre-MDT0001 complete [11216.441927] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 07:31:29 (1787398289) [11232.053503] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [11242.972145] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [11275.261898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.601176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11275.647167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11275.787857] Lustre: lustre-MDT0000: new disk, initializing [11275.912499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11275.936874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11280.461129] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11290.013703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11290.137990] Lustre: 349223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11290.185133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11290.188826] Lustre: Skipped 1 previous similar message [11290.252425] Lustre: lustre-MDT0001: new disk, initializing [11290.332648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11290.361095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11290.374996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11293.717896] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11302.441926] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11308.425420] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11315.376456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11315.653889] Lustre: lustre-OST0000: new disk, initializing [11315.660994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11315.672289] Lustre: 351191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11315.763873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11322.896042] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11323.451489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11323.473288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11323.591210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11332.995775] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11375.074099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11375.074529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.084924] LustreError: Skipped 1 previous similar message [11375.088646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11375.099215] Lustre: Skipped 1 previous similar message [11375.113173] Lustre: Skipped 1 previous similar message [11379.935536] Lustre: server umount lustre-OST0000 complete [11385.326158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11385.342778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11390.672191] Lustre: server umount lustre-MDT0000 complete [11394.197868] LustreError: 350017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398468 with bad export cookie 10700128035069432579 [11394.204540] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11394.212987] LustreError: 350017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11394.547189] Lustre: server umount lustre-MDT0001 complete [11403.699875] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11406.442196] Key type lgssc unregistered [11406.696135] LNet: 353371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11406.705064] LNetError: 353371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11406.729591] LNet: Removed LNI 192.168.206.112@tcp [11407.546646] Key type .llcrypt unregistered [11407.548808] Key type ._llcrypt unregistered [11487.229231] Key type ._llcrypt registered [11487.231122] Key type .llcrypt registered [11487.322445] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11487.600718] Key type .llcrypt unregistered [11487.603330] Key type ._llcrypt unregistered [11553.778954] Key type ._llcrypt registered [11553.781265] Key type .llcrypt registered [11553.859916] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11554.160545] Key type .llcrypt unregistered [11554.162586] Key type ._llcrypt unregistered [11583.092758] Key type ._llcrypt registered [11583.095396] Key type .llcrypt registered [11583.192540] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11583.507991] Key type .llcrypt unregistered [11583.512118] Key type ._llcrypt unregistered [11609.556818] Key type ._llcrypt registered [11609.559972] Key type .llcrypt registered [11609.658669] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11609.994311] Key type .llcrypt unregistered [11610.000826] Key type ._llcrypt unregistered [11678.575061] Key type ._llcrypt registered [11678.578826] Key type .llcrypt registered [11678.688251] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11679.026337] Key type .llcrypt unregistered [11679.030849] Key type ._llcrypt unregistered [11706.518764] Key type ._llcrypt registered [11706.520641] Key type .llcrypt registered [11706.628876] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11707.008618] Key type .llcrypt unregistered [11707.011677] Key type ._llcrypt unregistered [11738.862860] Key type ._llcrypt registered [11738.869119] Key type .llcrypt registered [11738.985199] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11739.357936] Key type .llcrypt unregistered [11739.359491] Key type ._llcrypt unregistered [11768.634417] Key type ._llcrypt registered [11768.636872] Key type .llcrypt registered [11768.768809] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11769.063909] Key type .llcrypt unregistered [11769.067044] Key type ._llcrypt unregistered [11793.862953] Key type ._llcrypt registered [11793.867836] Key type .llcrypt registered [11793.948662] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11794.263822] Key type .llcrypt unregistered [11794.265622] Key type ._llcrypt unregistered [11819.592671] Key type ._llcrypt registered [11819.597430] Key type .llcrypt registered [11819.694210] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11819.978762] Key type .llcrypt unregistered [11819.983338] Key type ._llcrypt unregistered [11878.516913] Key type ._llcrypt registered [11878.521693] Key type .llcrypt registered [11878.607846] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11878.856657] Key type .llcrypt unregistered [11878.858328] Key type ._llcrypt unregistered [11903.449200] Key type ._llcrypt registered [11903.453158] Key type .llcrypt registered [11903.591441] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11903.910485] Key type .llcrypt unregistered [11903.913076] Key type ._llcrypt unregistered [11930.950310] Key type ._llcrypt registered [11930.954382] Key type .llcrypt registered [11931.058368] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11931.274750] Key type .llcrypt unregistered [11931.276680] Key type ._llcrypt unregistered [11985.231529] Key type ._llcrypt registered [11985.234510] Key type .llcrypt registered [11985.380184] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [12002.479616] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [12003.630504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12003.735886] alg: No test for adler32 (adler32-zlib) [12004.885833] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [12005.147822] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [12006.824167] Key type lgssc registered [12007.901916] Lustre: Echo OBD driver; http://www.lustre.org/ [12044.095520] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12044.109403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12045.436940] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12045.479230] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12045.621071] Lustre: tst250fs-MDT0000: new disk, initializing [12045.686916] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12045.702476] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12050.957496] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12061.141071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12061.219826] Lustre: 373598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12061.289407] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12061.293600] Lustre: Skipped 1 previous similar message [12061.360655] Lustre: tst250fs-MDT0001: new disk, initializing [12061.458394] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12061.494832] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12061.503945] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12066.472277] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12071.592447] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12078.039563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12078.455787] Lustre: tst250fs-OST0000: new disk, initializing [12078.460971] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12078.472832] Lustre: 375232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12078.599949] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12081.864829] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12081.880447] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12081.998671] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12085.158529] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12096.848746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12097.045425] Lustre: tst250fs-OST0001: new disk, initializing [12097.048660] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12097.054316] Lustre: 376103:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12097.119209] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12103.436612] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12105.826058] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12105.835984] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12105.935312] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12109.834716] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12112.885464] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12112.889429] Lustre: Skipped 2 previous similar messages [12131.811101] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12131.827505] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12131.834133] Lustre: Skipped 3 previous similar messages [12131.850767] Lustre: Skipped 3 previous similar messages [12136.932116] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12136.957232] Lustre: Skipped 3 previous similar messages [12142.053655] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12142.067493] Lustre: Skipped 3 previous similar messages [12145.120129] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12145.282737] Lustre: server umount tst250fs-MDT0000 complete [12148.806116] LustreError: 373589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399223 with bad export cookie 9498258616745792743 [12148.810802] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12148.819956] LustreError: 373589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12149.339054] Lustre: server umount tst250fs-MDT0001 complete [12164.015246] Lustre: server umount tst250fs-OST0000 complete [12178.191374] Lustre: server umount tst250fs-OST0001 complete [12196.227482] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [12205.179307] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [12236.283385] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 07:48:29 (1787399309) [12237.764488] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12253.152169] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [12263.793256] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [12296.144291] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11992 sec ========= 07:49:29 (1787399369) [12298.170735] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:49:31 (1787399371) === [12302.217918] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:49:35 (1787399375) === [12344.726190] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [12347.986253] Key type lgssc unregistered [12348.382736] LNet: 385254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12348.392075] LNetError: 385254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12348.407188] LNet: Removed LNI 192.168.206.112@tcp [12349.252354] Key type .llcrypt unregistered [12349.262506] Key type ._llcrypt unregistered