[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511498079 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003130] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008406] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011134] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013040] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.022718] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026760] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029139] *** VALIDATE tmpfs *** [ 0.030511] *** VALIDATE proc *** [ 0.031203] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.033310] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036027] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040027] debug: unmapping init [mem 0xffffffffb7859000-0xffffffffb7860fff] [ 0.043151] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044699] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047009] ... generic registers: 4 [ 0.048019] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051011] ... event mask: 000000070000000f [ 0.052293] rcu: Hierarchical SRCU implementation. [ 0.054363] smp: Bringing up secondary CPUs ... [ 0.055553] x86: Booting SMP configuration: [ 0.056023] .... node #0, CPUs: #1 #2 #3 [ 0.061015] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137435] node 0 deferred pages initialised in 71ms [ 0.141094] devtmpfs: initialized [ 0.142224] x86/mm: Memory block size: 128MB [ 0.144779] gcov: version magic: 0x41383552 [ 0.146277] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148214] pinctrl core: initialized pinctrl subsystem [ 0.149173] [ 0.149644] ************************************************************* [ 0.150016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151011] ** ** [ 0.152011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153011] ** ** [ 0.154017] ** This means that this kernel is built to expose internal ** [ 0.155011] ** IOMMU data structures, which may compromise security on ** [ 0.156012] ** your system. ** [ 0.157013] ** ** [ 0.158013] ** If you see this message and you are not debugging the ** [ 0.159010] ** kernel, report this immediately to your vendor! ** [ 0.160011] ** ** [ 0.161012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162012] ************************************************************* [ 0.164337] NET: Registered protocol family 16 [ 0.165435] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173020] cpuidle: using governor menu [ 0.175180] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179686] PCI: Using configuration type 1 for base access [ 0.185123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209197] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.211028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.219070] cryptd: max_cpu_qlen set to 1000 [ 0.221285] ACPI: Added _OSI(Module Device) [ 0.222015] ACPI: Added _OSI(Processor Device) [ 0.223012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232924] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241124] ACPI: Interpreter enabled [ 0.242065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245013] ACPI: Using IOAPIC for interrupt routing [ 0.248092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256628] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269367] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.275029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.279076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.286269] acpiphp: Slot [2] registered [ 0.289142] acpiphp: Slot [5] registered [ 0.293103] acpiphp: Slot [6] registered [ 0.294000] acpiphp: Slot [7] registered [ 0.294095] acpiphp: Slot [8] registered [ 0.301158] acpiphp: Slot [9] registered [ 0.304134] acpiphp: Slot [10] registered [ 0.305120] acpiphp: Slot [3] registered [ 0.308146] acpiphp: Slot [4] registered [ 0.312279] acpiphp: Slot [11] registered [ 0.316089] acpiphp: Slot [12] registered [ 0.317107] acpiphp: Slot [13] registered [ 0.322443] acpiphp: Slot [14] registered [ 0.327349] acpiphp: Slot [15] registered [ 0.329152] acpiphp: Slot [16] registered [ 0.330836] acpiphp: Slot [17] registered [ 0.331173] acpiphp: Slot [18] registered [ 0.337115] acpiphp: Slot [19] registered [ 0.344818] acpiphp: Slot [20] registered [ 0.355096] acpiphp: Slot [21] registered [ 0.358163] acpiphp: Slot [22] registered [ 0.369221] acpiphp: Slot [23] registered [ 0.379151] acpiphp: Slot [24] registered [ 0.385145] acpiphp: Slot [25] registered [ 0.387541] acpiphp: Slot [26] registered [ 0.391196] acpiphp: Slot [27] registered [ 0.395205] acpiphp: Slot [28] registered [ 0.398120] acpiphp: Slot [29] registered [ 0.403129] acpiphp: Slot [30] registered [ 0.404000] acpiphp: Slot [31] registered [ 0.411085] PCI host bridge to bus 0000:00 [ 0.414018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.417026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.420031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.423026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.429022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.436030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.438647] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.454436] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.466484] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.504011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.516065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.524149] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.527016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.533017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.537624] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.544640] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.552041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.562079] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 16601 usecs [ 0.568820] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.577017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.605015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.613017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.625032] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.642015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.653000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.682014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.702000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.716014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.728000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.761014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.786000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.802014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.826016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.861016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.881000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.891014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.909079] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.942014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.963000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.978014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.988016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.020013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.049647] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.068013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.086016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.120016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.129000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.129000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.137000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.142424] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.147310] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.152000] iommu: Default domain type: Passthrough [ 1.156388] SCSI subsystem initialized [ 1.159168] ACPI: bus type USB registered [ 1.162140] usbcore: registered new interface driver usbfs [ 1.166017] usbcore: registered new interface driver hub [ 1.171030] usbcore: registered new device driver usb [ 1.176157] pps_core: LinuxPPS API ver. 1 registered [ 1.179012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.186058] PTP clock support registered [ 1.192037] EDAC MC: Ver: 3.0.0 [ 1.194125] PCI: Using ACPI for IRQ routing [ 1.195810] NetLabel: Initializing [ 1.202013] NetLabel: domain hash size = 128 [ 1.205013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.208074] NetLabel: unlabeled traffic allowed by default [ 1.214422] vgaarb: loaded [ 1.216288] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.218016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.239907] clocksource: Switched to clocksource kvm-clock [ 1.588405] VFS: Disk quotas dquot_6.6.0 [ 1.598483] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.612362] *** VALIDATE ramfs *** [ 1.619216] *** VALIDATE hugetlbfs *** [ 1.630661] pnp: PnP ACPI init [ 1.636428] pnp: PnP ACPI: found 6 devices [ 1.731460] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.742086] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.750570] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.758608] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.767455] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.774112] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.790837] NET: Registered protocol family 2 [ 1.799495] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.814292] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.835188] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.858493] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.874602] TCP: Hash tables configured (established 65536 bind 65536) [ 1.885651] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.894044] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.899427] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.903646] NET: Registered protocol family 1 [ 1.914707] RPC: Registered named UNIX socket transport module. [ 1.920458] RPC: Registered udp transport module. [ 1.925840] RPC: Registered tcp transport module. [ 1.930895] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.939361] NET: Registered protocol family 44 [ 1.944200] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.949937] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.953882] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.963094] PCI: CLS 0 bytes, default 64 [ 1.965390] Unpacking initramfs... [ 5.893407] debug: unmapping init [mem 0xffff8f28bcc54000-0xffff8f28bffbffff] [ 5.907390] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.913119] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.927123] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.925093] Initialise system trusted keyrings [ 7.933526] Key type blacklist registered [ 7.952319] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.980221] zbud: loaded [ 7.990533] *** VALIDATE nfs *** [ 7.993466] *** VALIDATE nfs4 *** [ 7.999727] pstore: using deflate compression [ 8.010333] Platform Keyring initialized [ 8.322394] NET: Registered protocol family 38 [ 8.325468] Key type asymmetric registered [ 8.327198] Asymmetric key parser 'x509' registered [ 8.330701] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 8.335612] io scheduler mq-deadline registered [ 8.337414] io scheduler kyber registered [ 8.338777] io scheduler bfq registered [ 8.341473] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 8.349250] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 8.353944] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 8.357572] ACPI: Power Button [PWRF] [ 8.364904] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.378379] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 8.405756] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 8.414775] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.441367] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.494535] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.550182] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.560577] Non-volatile memory driver v1.3 [ 8.565067] Linux agpgart interface v0.103 [ 8.647540] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 8.652505] vda: detected capacity change from 0 to 74760192 [ 8.689619] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.697413] vdb: detected capacity change from 0 to 1073741824 [ 8.757135] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.769422] vdc: detected capacity change from 0 to 2621440000 [ 8.813370] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.822282] vdd: detected capacity change from 0 to 2621440000 [ 8.875734] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.886258] vde: detected capacity change from 0 to 4294967296 [ 8.949747] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.956395] vdf: detected capacity change from 0 to 4294967296 [ 8.981390] libphy: Fixed MDIO Bus: probed [ 8.989663] usbcore: registered new interface driver usbserial_generic [ 8.995151] usbserial: USB Serial support registered for generic [ 8.998314] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.003616] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.005644] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.008556] mousedev: PS/2 mouse device common for all mice [ 9.013481] rtc_cmos 00:05: RTC can wake from S4 [ 9.017212] rtc_cmos 00:05: registered as rtc0 [ 9.019405] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.024025] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.027454] intel_pstate: CPU model not supported [ 9.032392] hid: raw HID events driver (C) Jiri Kosina [ 9.033687] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.040042] usbcore: registered new interface driver usbhid [ 9.040049] usbhid: USB HID core driver [ 9.040159] drop_monitor: Initializing network drop monitor service [ 9.040290] Initializing XFRM netlink socket [ 9.040650] NET: Registered protocol family 10 [ 9.043372] Segment Routing with IPv6 [ 9.047134] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.069591] NET: Registered protocol family 17 [ 9.072094] mpls_gso: MPLS GSO support [ 9.084838] RAS: Correctable Errors collector initialized. [ 9.092036] AVX version of gcm_enc/dec engaged. [ 9.094478] AES CTR mode by8 optimization enabled [ 9.323816] sched_clock: Marking stable (9323796686, 0)->(10497286605, -1173489919) [ 9.331987] registered taskstats version 1 [ 9.338494] Loading compiled-in X.509 certificates [ 9.341471] zswap: loaded using pool lzo/zbud [ 9.398856] Key type big_key registered [ 9.418868] Key type encrypted registered [ 9.423291] ima: No TPM chip found, activating TPM-bypass! [ 9.431674] ima: Allocated hash algorithm: sha1 [ 9.433250] ima: No architecture policies found [ 9.441459] evm: Initialising EVM extended attributes: [ 9.447995] evm: security.selinux [ 9.453928] evm: security.ima [ 9.457844] evm: security.capability [ 9.463726] evm: HMAC attrs: 0x1 [ 9.471607] rtc_cmos 00:05: setting system clock to 2026-08-31 04:58:21 UTC (1788152301) [ 9.486673] debug: unmapping init [mem 0xffffffffb8803000-0xffffffffb89fffff] [ 9.496186] debug: unmapping init [mem 0xffffffffb7582000-0xffffffffb7858fff] [ 9.506157] Write protecting the kernel read-only data: 28672k [ 9.513524] debug: unmapping init [mem 0xffffffffb5c03000-0xffffffffb5dfffff] [ 9.523368] debug: unmapping init [mem 0xffffffffb6514000-0xffffffffb65fffff] [ 9.596163] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.625534] systemd[1]: Detected virtualization kvm. [ 9.631678] systemd[1]: Detected architecture x86-64. [ 9.639330] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.688592] systemd[1]: No hostname configured. [ 9.691988] systemd[1]: Set hostname to . [ 9.695541] random: systemd: uninitialized urandom read (16 bytes read) [ 9.700343] systemd[1]: Initializing machine ID from random generator. [ 9.837905] random: ln: uninitialized urandom read (6 bytes read) [ 9.993296] random: systemd: uninitialized urandom read (16 bytes read) [ 9.996579] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 10.007550] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 10.015604] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.661922] device-mapper: uevent: version 1.0.3 [ 11.666120] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [ 13.478266] random: fast init done ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.548519] virtio_net virtio0 ens2: renamed from eth0 [ 13.832996] scsi host0: ata_piix [ 13.905606] scsi host1: ata_piix [ 13.906673] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.916447] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.989809] random: crng init done [ 18.994395] random: 7 urandom warning(s) missed due to ratelimiting [ 19.859739] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 21.786517] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ 21.849058] hrtimer: interrupt took 3328977 ns [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.678847] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.463251] SELinux: Disabled at runtime. [ 25.570098] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.579873] systemd[1]: Detected virtualization kvm. [ 25.582645] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.240920] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.247880] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.261779] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.277949] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.284419] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.344667] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.366968] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. Startin[ 27.752292] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g Apply Kernel Variables... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 28.974230] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.789057] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.865278] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.220522] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.284284] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 35.526218] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 36.445435] NFS: Registering the id_resolver key type [ 36.456708] Key type id_resolver registered [ 36.458347] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg646-server login: [ 104.349464] libcfs: loading out-of-tree module taints kernel. [ 104.422556] Key type ._llcrypt registered [ 104.424436] Key type .llcrypt registered [ 104.561210] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 125.101931] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 126.814598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 126.833970] alg: No test for adler32 (adler32-zlib) [ 128.341422] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 129.186266] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 130.920219] Key type lgssc registered [ 132.630758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 151.862458] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 194.068374] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 207.698176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 207.730637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.107145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.157044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.231442] Lustre: lustre-MDT0000: new disk, initializing [ 209.310294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.322926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.824564] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.212700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.389238] Lustre: 6529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 228.430600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 228.440332] Lustre: Skipped 1 previous similar message [ 228.587200] Lustre: lustre-MDT0001: new disk, initializing [ 228.645815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.670727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 228.682531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.504631] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.373549] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 247.788895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.022226] Lustre: lustre-OST0000: new disk, initializing [ 248.027249] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.034154] Lustre: 8466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.101477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 254.480461] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.002062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 255.018896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 255.083967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 267.958728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 268.105435] Lustre: lustre-OST0001: new disk, initializing [ 268.114154] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 268.127213] Lustre: 9537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 268.200501] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 274.048955] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.011588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.024855] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.071956] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 286.525987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.939917] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 302.629847] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing check_logdir /tmp/testlogs/ [ 308.175986] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing yml_node [ 312.912762] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 316.011917] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 318.845169] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 320.464364] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 01:03:31 EDT 2026 [ 338.105487] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 339.532163] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 356.834845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 356.853061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.862733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.906254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.907689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.927028] Lustre: Skipped 1 previous similar message [ 359.940320] Lustre: Skipped 3 previous similar messages [ 362.412411] Lustre: server umount lustre-MDT0000 complete [ 366.237404] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152658 with bad export cookie 14780687203732603987 [ 366.241307] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.246267] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 370.145693] LustreError: 6534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.149505] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.161628] LustreError: 6534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 370.196436] Lustre: Skipped 1 previous similar message [ 370.209909] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.218365] Lustre: Skipped 1 previous similar message [ 371.183701] LustreError: 6534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.210755] LustreError: 6534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 372.692491] Lustre: server umount lustre-MDT0001 complete [ 377.349933] Lustre: server umount lustre-OST0000 complete [ 381.108748] Lustre: server umount lustre-OST0001 complete [ 387.213803] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 398.571145] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 436.367335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.605610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 436.642911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 436.739742] Lustre: lustre-MDT0000: new disk, initializing [ 436.816749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.829928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 440.977237] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 451.774779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.950300] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 452.019140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 452.025913] Lustre: Skipped 1 previous similar message [ 452.139260] Lustre: lustre-MDT0001: new disk, initializing [ 452.255782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 452.313993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 452.321835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 456.498864] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 466.033462] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 471.558151] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 478.528641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 478.844608] Lustre: lustre-OST0000: new disk, initializing [ 478.849470] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 478.858260] Lustre: 17058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 478.972772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.883909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 480.899076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 481.048096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 485.414880] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 495.721206] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 500.834117] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 501.070585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 506.114713] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 506.292367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 511.981477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 511.989110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.994612] LustreError: Skipped 1 previous similar message [ 512.023713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.032526] Lustre: Skipped 2 previous similar messages [ 512.057569] Lustre: Skipped 3 previous similar messages [ 514.494931] Lustre: server umount lustre-OST0000 complete [ 518.626394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 518.645154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.664858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.228303] Lustre: server umount lustre-MDT0000 complete [ 527.333621] LustreError: 15104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.354546] LustreError: 15104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 528.118282] LustreError: 17057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152820 with bad export cookie 14780687203732605520 [ 528.125665] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.130830] LustreError: 17057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 528.476561] Lustre: server umount lustre-MDT0001 complete [ 534.417065] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:07:05 (1788152825) [ 550.789198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.132286] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.195417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.775521] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 556.526293] LustreError: 19934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.873513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.378530] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 574.887573] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 580.796858] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 582.859296] Lustre: Failing over lustre-MDT0000 [ 584.675360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 584.688858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.710202] Lustre: Skipped 1 previous similar message [ 584.715032] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.740435] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 585.078993] Lustre: server umount lustre-MDT0000 complete [ 602.083269] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788152878/real 1788152878] req@ffff8f293a3baa00 x1875013713165184/t0(0) o400->MGC192.168.206.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788152894 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 602.104790] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 602.123098] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.148622] LustreError: 19933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 604.765489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.325404] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcd1f8cb34aa4cfd6 [ 612.337743] Lustre: MGC192.168.206.146@tcp: Connection restored to 0@lo (at 0@lo) [ 612.672935] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 616.437470] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 617.955975] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 617.963691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 617.983958] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 625.907759] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 632.289514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.295833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.300986] Lustre: Skipped 2 previous similar messages [ 632.304704] Lustre: Skipped 2 previous similar messages [ 635.909542] Lustre: server umount lustre-MDT0000 complete [ 639.078692] LustreError: 22593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152931 with bad export cookie 14780687203732606934 [ 639.086404] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.094039] LustreError: 22593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 639.367542] Lustre: server umount lustre-MDT0001 complete [ 647.591034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.923945] LustreError: 23269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.943233] LustreError: 23269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 648.028153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 648.035491] Lustre: Skipped 2 previous similar messages [ 651.950174] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 660.174727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 665.146309] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 671.563873] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 677.060981] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 679.301424] Lustre: Failing over lustre-MDT0000 [ 679.576286] Lustre: server umount lustre-MDT0000 complete [ 680.929766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 680.939641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.999975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.189899] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 699.593183] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 703.306436] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 704.995255] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 704.999389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 705.006380] Lustre: Skipped 1 previous similar message [ 705.026273] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 711.829945] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 719.333159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 719.338733] Lustre: Skipped 3 previous similar messages [ 721.845303] Lustre: server umount lustre-MDT0000 complete [ 724.453778] LustreError: 23270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 724.486487] LustreError: 23270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 725.034115] LustreError: 23249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153017 with bad export cookie 14780687203732607872 [ 725.039940] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.049206] LustreError: 23249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 725.316494] Lustre: server umount lustre-MDT0001 complete [ 734.303547] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:10:25 (1788153025) [ 749.379338] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 761.350454] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 798.596476] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 808.506861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.708301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 808.734879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 808.854077] Lustre: lustre-MDT0000: new disk, initializing [ 808.942780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.945879] Lustre: Skipped 2 previous similar messages [ 808.957819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 812.926429] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 823.440602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.583749] Lustre: 29645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 823.883951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 828.611578] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 838.059604] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 843.294228] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 850.137481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 850.450799] Lustre: 31613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 852.206684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 852.247285] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 852.325080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 852.352727] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 856.705781] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 865.989809] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 871.149094] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 893.412228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 893.412974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.423794] LustreError: Skipped 1 previous similar message [ 893.438257] Lustre: Skipped 3 previous similar messages [ 893.438926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 893.454757] Lustre: Skipped 1 previous similar message [ 896.862328] Lustre: server umount lustre-OST0000 complete [ 906.138951] Lustre: server umount lustre-MDT0000 complete [ 908.769712] LustreError: 29650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.796093] LustreError: 29650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 909.701889] LustreError: 29636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153201 with bad export cookie 14780687203732608278 [ 909.705237] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 909.709042] LustreError: 29636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 910.284770] Lustre: server umount lustre-MDT0001 complete [ 921.555226] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 924.258278] Key type lgssc unregistered [ 924.650423] LNet: 33589:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.662295] LNetError: 33589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.691694] LNet: Removed LNI 192.168.206.146@tcp [ 925.998156] Key type .llcrypt unregistered [ 926.002133] Key type ._llcrypt unregistered [ 951.763298] Key type ._llcrypt registered [ 951.768359] Key type .llcrypt registered [ 951.860864] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 953.110407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 953.138746] alg: No test for adler32 (adler32-zlib) [ 954.242672] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 954.601070] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 956.296986] Key type lgssc registered [ 957.512989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 964.972564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 964.995141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 966.387382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 970.620848] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 976.866514] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.888120] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 979.391922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.752978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.659926] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 992.322536] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 998.008140] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1004.943859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1005.313221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.051591] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1011.473199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1017.735564] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1024.754592] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1032.731991] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:15:24 (1788153324) [ 1034.380664] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1036.147104] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1042.117944] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:15:33 (1788153333) [ 1049.078970] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:15:40 (1788153340) [ 1049.983306] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1058.362888] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1058.373312] Lustre: Skipped 2 previous similar messages [ 1059.970402] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:15:51 (1788153351) [ 1063.235615] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1063.238233] Lustre: Skipped 3 previous similar messages [ 1080.142616] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1080.146461] Lustre: Skipped 6 previous similar messages [ 1082.508086] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:16:13 (1788153373) [ 1084.877357] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1088.274240] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1095.459455] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:16:26 (1788153386) [ 1097.032383] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1097.037389] Lustre: Skipped 3 previous similar messages [ 1108.431749] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:16:39 (1788153399) [ 1122.278227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.278768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1122.287123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1127.278572] Lustre: server umount lustre-OST0000 complete [ 1132.514956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.531646] Lustre: Skipped 1 previous similar message [ 1132.537238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1132.550650] Lustre: Skipped 1 previous similar message [ 1137.066590] Lustre: server umount lustre-MDT0000 complete [ 1137.635854] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.665408] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1141.353037] LustreError: 34982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153433 with bad export cookie 9184570983908682746 [ 1141.365837] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1141.374171] LustreError: 34982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1141.911497] Lustre: server umount lustre-MDT0001 complete [ 1152.942989] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1155.579751] Key type lgssc unregistered [ 1155.913796] LNet: 42853:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.941171] LNetError: 42853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.961983] LNet: Removed LNI 192.168.206.146@tcp [ 1156.739439] Key type .llcrypt unregistered [ 1156.742681] Key type ._llcrypt unregistered [ 1173.371958] Key type ._llcrypt registered [ 1173.376789] Key type .llcrypt registered [ 1173.505857] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 1189.767686] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1191.191929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1191.311317] alg: No test for adler32 (adler32-zlib) [ 1192.475176] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1192.765199] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1194.480173] Key type lgssc registered [ 1195.584824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1234.315251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1234.369154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.718642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1235.754194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1235.879717] Lustre: lustre-MDT0000: new disk, initializing [ 1235.958247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.977884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1240.842694] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1251.683069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.837870] Lustre: 46005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1251.913501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1251.919116] Lustre: Skipped 1 previous similar message [ 1252.019329] Lustre: lustre-MDT0001: new disk, initializing [ 1252.153408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1252.225253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1252.240347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1257.988684] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1268.883614] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1275.626690] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1283.928691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.291628] Lustre: lustre-OST0000: new disk, initializing [ 1284.300377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1284.310354] Lustre: 47975:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1284.453182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.338027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1291.350745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1291.443742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1291.709679] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1301.754695] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1305.782165] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1306.101356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1310.761657] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1310.907177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1317.349440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.351639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1317.370592] Lustre: Skipped 1 previous similar message [ 1317.379183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1317.385513] Lustre: Skipped 1 previous similar message [ 1322.470379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1322.478495] Lustre: Skipped 1 previous similar message [ 1327.074753] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1327.262141] Lustre: server umount lustre-OST0000 complete [ 1332.706900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.709189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1332.722371] Lustre: Skipped 1 previous similar message [ 1332.735507] Lustre: Skipped 1 previous similar message [ 1337.068390] Lustre: server umount lustre-MDT0000 complete [ 1340.121699] LustreError: 46000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153632 with bad export cookie 5851425870745690789 [ 1340.136375] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.517897] Lustre: server umount lustre-MDT0001 complete [ 1348.960331] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:20:39 (1788153639) [ 1355.046158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.391517] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.428630] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1355.460940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1360.232359] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1360.870446] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.987839] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.706168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.032704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1373.389965] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1380.708961] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1386.232772] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1393.149459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1393.444954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.683784] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1405.661447] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1409.869549] LustreError: 52968:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1410.617192] Lustre: Modifying parameter general.timeout=1 in log params [ 1412.478405] Lustre: Modifying parameter general.timeout=3 in log params [ 1412.484055] Lustre: Skipped 1 previous similar message [ 1415.096549] Lustre: Modifying parameter general.timeout=6 in log params [ 1415.099587] Lustre: Skipped 2 previous similar messages [ 1419.123953] Lustre: Modifying parameter general.timeout=11 in log params [ 1419.127468] Lustre: Skipped 4 previous similar messages [ 1427.180421] Lustre: Modifying parameter general.timeout=21 in log params [ 1427.184255] Lustre: Skipped 9 previous similar messages [ 1443.731586] Lustre: Modifying parameter general.timeout=41 in log params [ 1443.735552] Lustre: Skipped 19 previous similar messages [ 1476.181540] Lustre: Modifying parameter general.timeout=81 in log params [ 1476.192680] Lustre: Skipped 39 previous similar messages [ 1526.528689] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:23:37 (1788153817) [ 1529.215730] Lustre: 59899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1531.101961] Lustre: 60010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1534.726619] Lustre: 60208:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1534.748227] Lustre: 60208:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1541.380625] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 in log params [ 1541.384330] Lustre: Skipped 57 previous similar messages [ 1565.426930] Lustre: 61766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1565.433796] Lustre: 61766:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1565.453541] LustreError: 61766:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1570.833860] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:24:21 (1788153861) [ 1587.683621] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1587.689906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1587.702485] Lustre: Skipped 1 previous similar message [ 1587.723368] Lustre: Skipped 2 previous similar messages [ 1590.589036] Lustre: server umount lustre-MDT0000 complete [ 1592.823197] LustreError: 51681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1592.848501] LustreError: 51681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1594.287619] LustreError: 50418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153886 with bad export cookie 5851425870745691671 [ 1594.291977] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1594.302408] LustreError: 50418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1594.677679] Lustre: server umount lustre-MDT0001 complete [ 1609.010154] Lustre: server umount lustre-OST0000 complete [ 1634.157123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.387508] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1634.580987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.114678] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1647.849403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.918816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1647.944580] Lustre: 64303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1652.369114] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1659.901534] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1666.023427] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1673.598891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1673.862289] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1674.316610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.326778] Lustre: Skipped 1 previous similar message [ 1680.152507] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1687.298703] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1689.379985] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1689.391203] Lustre: Skipped 14 previous similar messages [ 1689.433974] Lustre: 66870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1698.788701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1698.800182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.809311] Lustre: Skipped 1 previous similar message [ 1698.813324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1702.484901] Lustre: server umount lustre-OST0000 complete [ 1709.538381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.543648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1709.553840] Lustre: Skipped 1 previous similar message [ 1709.554833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.554839] Lustre: Skipped 2 previous similar messages [ 1709.583308] LustreError: Skipped 1 previous similar message [ 1712.600369] Lustre: server umount lustre-MDT0000 complete [ 1714.657328] LustreError: 64311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.678319] LustreError: 64311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1716.071056] LustreError: 65054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154008 with bad export cookie 5851425870745694639 [ 1716.072615] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1716.080490] LustreError: 65054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1716.386728] Lustre: server umount lustre-MDT0001 complete [ 1726.382717] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 1728.869745] Key type lgssc unregistered [ 1729.088623] LNet: 68079:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.093231] LNetError: 68079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.115052] LNet: Removed LNI 192.168.206.146@tcp [ 1729.935733] Key type .llcrypt unregistered [ 1729.938314] Key type ._llcrypt unregistered [ 1759.652208] Key type ._llcrypt registered [ 1759.654358] Key type .llcrypt registered [ 1759.756294] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 1761.099801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1761.234752] alg: No test for adler32 (adler32-zlib) [ 1762.419211] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1762.731793] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 1764.440151] Key type lgssc registered [ 1765.489825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1772.726763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1772.758600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1774.148282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.027639] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1784.293676] LustreError: 69491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1784.310321] LustreError: 69491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1786.713918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.466444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1791.707296] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1798.246781] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1803.989505] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1810.830365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1811.296632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.135228] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1824.323553] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1832.147702] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1843.687507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1843.699169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.725886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1845.219784] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1845.220581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1845.250532] Lustre: Skipped 2 previous similar messages [ 1847.147768] Lustre: server umount lustre-MDT0000 complete [ 1850.341689] LustreError: 69492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1850.355480] LustreError: 69492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1850.421727] LustreError: 72178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154142 with bad export cookie 2387288344871318115 [ 1850.432539] LustreError: 72178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1850.432803] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1850.809551] Lustre: server umount lustre-MDT0001 complete [ 1864.482786] Lustre: server umount lustre-OST0000 complete [ 1892.313563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1892.687377] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1892.715606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1893.051734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.695249] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1910.343692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.527089] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1910.589351] Lustre: 74363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1910.951464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1915.780870] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1922.449330] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1928.445341] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1937.188679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1937.394457] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1937.793267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.545554] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 1953.386915] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1955.450333] Lustre: 76941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1957.678254] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1966.561886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1966.575320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.584766] Lustre: Skipped 1 previous similar message [ 1966.593801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1970.156477] Lustre: server umount lustre-OST0000 complete [ 1977.320294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.331197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1977.332087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1977.332092] Lustre: Skipped 2 previous similar messages [ 1977.341160] Lustre: Skipped 2 previous similar messages [ 1977.354413] LustreError: Skipped 1 previous similar message [ 1980.502222] Lustre: server umount lustre-MDT0000 complete [ 1982.434330] LustreError: 74375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.452605] LustreError: 74375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1984.398351] LustreError: 74354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154276 with bad export cookie 2387288344871318948 [ 1984.407774] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1984.422331] LustreError: 74354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1984.846597] Lustre: server umount lustre-MDT0001 complete [ 1997.278097] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 2000.069299] Key type lgssc unregistered [ 2000.436492] LNet: 78152:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.443943] LNetError: 78152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.474897] LNet: Removed LNI 192.168.206.146@tcp [ 2001.504514] Key type .llcrypt unregistered [ 2001.509428] Key type ._llcrypt unregistered [ 2028.729266] Key type ._llcrypt registered [ 2028.731615] Key type .llcrypt registered [ 2028.824690] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2029.736837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2029.834969] alg: No test for adler32 (adler32-zlib) [ 2030.836501] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2031.049210] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 2032.680122] Key type lgssc registered [ 2033.508438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2040.376715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.405447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.870650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.524230] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2051.043700] LustreError: 79568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.061134] LustreError: 79568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2054.201183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.755372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2058.727413] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2066.930362] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2074.104126] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2081.264557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2082.012123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2089.077462] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2097.762993] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2103.946326] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2104.938183] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2105.989599] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2108.494277] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2108.499421] Lustre: Skipped 2 previous similar messages [ 2113.404205] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2113.409457] Lustre: Skipped 4 previous similar messages [ 2122.192967] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2122.199984] Lustre: Skipped 8 previous similar messages [ 2138.256843] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2138.262353] Lustre: Skipped 16 previous similar messages [ 2170.327855] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2170.332515] Lustre: Skipped 35 previous similar messages [ 2201.549211] Lustre: 87664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2204.247583] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:34:55 (1788154495) [ 2205.613546] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2207.342650] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:34:58 (1788154498) [ 2212.322955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2212.328775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.342413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2212.351055] Lustre: Skipped 1 previous similar message [ 2217.448160] LustreError: 81395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.621954] Lustre: server umount lustre-OST0000 complete [ 2222.561719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.572667] Lustre: Skipped 1 previous similar message [ 2222.576371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.580664] Lustre: Skipped 1 previous similar message [ 2227.684158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.690408] Lustre: Skipped 1 previous similar message [ 2232.805320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2232.813339] Lustre: Skipped 1 previous similar message [ 2235.360122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2235.488701] Lustre: server umount lustre-MDT0000 complete [ 2237.924024] LustreError: 79564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2237.948686] LustreError: 79564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2239.376919] LustreError: 80311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154531 with bad export cookie 16351912607417824249 [ 2239.377588] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.389899] LustreError: 80311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2239.729332] Lustre: server umount lustre-MDT0001 complete [ 2249.240654] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 2252.089184] Key type lgssc unregistered [ 2252.437555] LNet: 88916:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.452202] LNetError: 88916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.467890] LNet: Removed LNI 192.168.206.146@tcp [ 2253.528289] Key type .llcrypt unregistered [ 2253.531373] Key type ._llcrypt unregistered [ 2258.354137] Key type ._llcrypt registered [ 2258.356629] Key type .llcrypt registered [ 2258.483598] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2263.845419] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules [ 2264.869192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2264.909571] alg: No test for adler32 (adler32-zlib) [ 2265.950510] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2266.211434] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 2267.922326] Key type lgssc registered [ 2269.467687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2270.752755] LustreError: 90160:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2287.763263] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2293.285268] LustreError: 90160:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2295.979789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2296.009108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.601393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2301.129673] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2307.045518] LustreError: 91192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.071128] LustreError: 91192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2311.557690] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:36:41 (1788154601) [ 2312.179466] LustreError: 91191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.287707] LustreError: 91192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.550415] LustreError: 91224:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2318.569449] LustreError: 91224:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2318.580972] LustreError: 91224:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2319.006377] Lustre: server umount lustre-MDT0000 complete [ 2333.679644] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 2336.661367] Key type lgssc unregistered [ 2336.846435] LNet: 92719:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.854841] LNetError: 92719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.875240] LNet: Removed LNI 192.168.206.146@tcp [ 2337.929662] Key type .llcrypt unregistered [ 2337.933833] Key type ._llcrypt unregistered [ 2356.954738] Key type ._llcrypt registered [ 2356.959520] Key type .llcrypt registered [ 2357.146531] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2358.234472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2358.323500] alg: No test for adler32 (adler32-zlib) [ 2359.353424] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2359.690656] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 2361.368329] Key type lgssc registered [ 2362.487815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2369.375761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2369.411607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.917939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.608979] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2380.259984] LustreError: 93940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.279586] LustreError: 93940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2385.383870] LustreError: 93941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.689652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.474266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.393676] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2399.845927] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2406.044893] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2413.698255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2414.491209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2422.049243] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2430.270085] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2482.927228] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:39:33 (1788154773) [ 2484.893085] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2486.798375] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:39:37 (1788154777) [ 2493.421507] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.423604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2493.453805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2497.683145] Lustre: server umount lustre-MDT0000 complete [ 2498.533532] LustreError: 96810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.555168] LustreError: 96810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2501.909207] LustreError: 93922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154793 with bad export cookie 4851557107002723143 [ 2501.917290] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2501.931521] LustreError: 93922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2502.292076] Lustre: server umount lustre-MDT0001 complete [ 2517.071721] Lustre: server umount lustre-OST0000 complete [ 2528.628844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.910338] LustreError: 98236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.061522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.064881] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2534.377227] LustreError: 98236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.392093] LustreError: 98236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2542.028468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.683556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2547.074565] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2554.362963] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2559.313042] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2572.402219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2572.632661] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2572.642427] LustreError: 98225:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2572.684137] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2572.707739] LustreError: 100319:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -98 [ 2572.723535] LustreError: 100319:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 2572.730500] LustreError: 100319:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2572.817589] Lustre: server umount lustre-OST0000 complete [ 2572.823091] LustreError: 100319:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2581.585787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2581.831501] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2581.836887] LustreError: 98225:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2581.853057] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2581.860555] LustreError: 100785:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -98 [ 2581.867584] LustreError: 100785:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 2581.870996] LustreError: 100785:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2581.952393] Lustre: server umount lustre-OST0000 complete [ 2581.959621] LustreError: 100785:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2591.205668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2591.331099] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2591.338474] Lustre: Found index 0 for lustre-OST0000, updating log [ 2591.350734] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2591.423362] Lustre: lustre-OST0000: new disk, initializing [ 2591.427259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2591.436183] Lustre: 101320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2591.692703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2597.845936] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2601.402710] Lustre: Failing over lustre-OST0000 [ 2601.605123] Lustre: server umount lustre-OST0000 complete [ 2609.125447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.136069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2609.138771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.138790] Lustre: Skipped 2 previous similar messages [ 2609.143487] Lustre: Skipped 3 previous similar messages [ 2612.258487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2612.274417] Lustre: Skipped 2 previous similar messages [ 2615.497708] Lustre: server umount lustre-MDT0000 complete [ 2617.313758] LustreError: 98967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.334818] LustreError: 98967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2619.237599] LustreError: 100802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154911 with bad export cookie 4851557107002725894 [ 2619.240507] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2619.247382] LustreError: 100802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2619.512281] Lustre: server umount lustre-MDT0001 complete [ 2628.421960] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:41:59 (1788154919) [ 2644.913551] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2655.108788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2655.716817] LustreError: 103666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.893881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.642304] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2670.010751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.872642] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2678.937824] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2685.954100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2686.667460] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2693.396235] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2695.850960] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2695.856308] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2695.882626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2700.011831] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:05 [ 2700.031354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2700.041606] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2700.043139] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2700.052563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2701.908935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2702.069763] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2702.187690] Lustre: lustre-OST0001: new disk, initializing [ 2702.190937] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2702.199841] Lustre: 105988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2702.542199] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2702.551311] Lustre: Skipped 2 previous similar messages [ 2708.066375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2708.071901] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2708.229865] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2708.993721] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2717.029595] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2728.290237] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2732.513935] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2732.523284] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.549031] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2736.972530] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.46@tcp (stopping) [ 2736.977079] Lustre: Skipped 3 previous similar messages [ 2738.656466] Lustre: server umount lustre-MDT0001 complete [ 2739.173685] LustreError: 103667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.182204] LustreError: 103667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2749.451853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.638393] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2749.643800] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2749.672882] Lustre: 103656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2751.413334] LustreError: 107565:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2751.422545] LustreError: 107565:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2751.439584] LustreError: 103658:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2751.545739] LustreError: 107571:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.146@tcp: cfg command failed: rc = -17 [ 2751.557841] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2751.557841] [ 2753.668653] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 2755.562329] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2762.929419] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:44:13 (1788155053) [ 2780.176641] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 2799.721546] Lustre: 109603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2824.891672] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2828.665604] Lustre: 110738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2917.350050] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.369912] Lustre: Skipped 2 previous similar messages [ 2917.378830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2917.390269] Lustre: Skipped 3 previous similar messages [ 2921.413459] Lustre: server umount lustre-MDT0000 complete [ 2922.466209] LustreError: 103661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2922.488134] LustreError: 103661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 2925.126616] LustreError: 103646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155217 with bad export cookie 4851557107002726895 [ 2925.131543] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2925.137222] LustreError: 103646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2925.495783] Lustre: server umount lustre-MDT0001 complete [ 2939.209318] Lustre: server umount lustre-OST0000 complete [ 2943.456107] Lustre: 93250:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155219/real 1788155219] req@ffff8f280fd08000 x1875016051692416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155235 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2943.489118] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.496843] Lustre: Skipped 2 previous similar messages [ 2943.869532] Lustre: server umount lustre-OST0001 complete [ 2949.840055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2960.655391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2969.822928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2980.804425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3002.835849] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3013.675677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.695448] Lustre: lustre-MDT0000: reset Object Index mappings [ 3014.250563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.260479] Lustre: Skipped 1 previous similar message [ 3019.077802] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3028.346847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3028.373205] Lustre: lustre-MDT0001: reset Object Index mappings [ 3034.267671] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3037.329190] Lustre: 115839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3045.141160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3053.109663] LustreError: 116193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.131466] LustreError: 116193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3053.151268] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3053.544858] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3059.206740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3062.466654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3068.414466] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3068.421972] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3068.977691] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3077.429397] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3081.356543] Lustre: 117716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3137.559719] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:50:28 (1788155428) [ 3165.672275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.684183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3165.684361] Lustre: Skipped 2 previous similar messages [ 3165.700531] Lustre: Skipped 3 previous similar messages [ 3169.673196] Lustre: server umount lustre-MDT0000 complete [ 3173.248549] LustreError: 114670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155465 with bad export cookie 4851557107002774012 [ 3173.252427] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3173.258481] LustreError: 114670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3179.905961] Lustre: server umount lustre-MDT0001 complete [ 3190.162735] Lustre: server umount lustre-OST0000 complete [ 3200.858557] Lustre: server umount lustre-OST0001 complete [ 3209.471673] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 3221.115506] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3261.666222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.948519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3261.976134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3262.081347] Lustre: lustre-MDT0000: new disk, initializing [ 3262.178831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.182520] Lustre: Skipped 3 previous similar messages [ 3262.211106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3266.496544] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3273.252823] Lustre: server umount lustre-MDT0000 complete [ 3284.077641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3284.261363] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3284.269438] Lustre: 121987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3284.276570] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3289.728178] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3296.019903] Lustre: server umount lustre-MDT0000 complete [ 3306.197910] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:53:17 (1788155597) [ 3307.644194] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3309.623777] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:53:20 (1788155600) [ 3328.621107] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 3337.738980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.464434] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3348.653237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.763824] Lustre: 124040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3348.818768] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3348.821505] Lustre: Skipped 1 previous similar message [ 3348.889769] Lustre: lustre-MDT0001: new disk, initializing [ 3349.024109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3349.034534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3352.825969] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3357.189138] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3364.104979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3364.321465] Lustre: lustre-OST0000: new disk, initializing [ 3364.325233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3364.330207] Lustre: 125614:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3366.393828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3366.403206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3366.540388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3371.258820] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3383.934437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.126886] Lustre: lustre-OST0001: new disk, initializing [ 3384.134094] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3384.139716] Lustre: 126486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3386.123105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3386.130585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3386.307511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3390.199176] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 3399.071446] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3402.301282] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3814.837419] Lustre: 132920:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3814.849777] Lustre: 132920:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3814.853577] Lustre: 132920:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3814.858214] Lustre: 132920:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3814.865457] Lustre: 132920:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3814.871828] Lustre: 132920:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4137.839855] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:07:07 (1788156427) [ 4154.340810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.349102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4154.357423] Lustre: Skipped 4 previous similar messages [ 4154.360622] Lustre: Skipped 4 previous similar messages [ 4159.465825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4159.476276] Lustre: Skipped 3 previous similar messages [ 4168.160125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4168.426189] Lustre: server umount lustre-MDT0000 complete [ 4169.708558] LustreError: 127304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4169.742248] LustreError: 127304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 4172.423034] LustreError: 132964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156464 with bad export cookie 4851557107002843564 [ 4172.433129] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4172.447286] LustreError: 132964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4174.820550] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4174.829372] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4174.842031] Lustre: Skipped 3 previous similar messages [ 4174.846383] Lustre: Skipped 4 previous similar messages [ 4178.987778] Lustre: server umount lustre-MDT0001 complete [ 4188.523869] Lustre: server umount lustre-OST0000 complete [ 4198.578381] Lustre: server umount lustre-OST0001 complete [ 4205.374563] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 4213.134082] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 4251.779174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.022972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4252.055171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4252.172272] Lustre: lustre-MDT0000: new disk, initializing [ 4252.282993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.288421] Lustre: Skipped 5 previous similar messages [ 4252.314438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4257.567817] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 4268.804492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4268.948963] Lustre: 135980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4269.018632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4269.024506] Lustre: Skipped 1 previous similar message [ 4269.079261] Lustre: lustre-MDT0001: new disk, initializing [ 4269.210302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4269.241379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4273.079091] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 4281.829742] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4287.440563] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4294.009332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4294.410853] Lustre: lustre-OST0000: new disk, initializing [ 4294.417273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4294.427131] Lustre: 137948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4294.558334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.561976] Lustre: Skipped 1 previous similar message [ 4296.254165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4296.269455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4296.407238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4301.470542] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 4310.712849] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4320.192222] Lustre: lustre-MDD0000: changelog on [ 4322.883539] Lustre: lustre-MDD0001: changelog on [ 6402.766460] Lustre: lustre-MDD0001: changelog off [ 6405.143845] Lustre: 135987:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6406.252654] Lustre: lustre-MDD0000: changelog off [ 6416.367055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6416.376570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.398855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6416.406315] Lustre: Skipped 4 previous similar messages [ 6419.007686] Lustre: server umount lustre-OST0000 complete [ 6424.545381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6424.553602] LustreError: Skipped 1 previous similar message [ 6424.560810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6424.575307] Lustre: Skipped 1 previous similar message [ 6424.581365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6426.605749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6430.312371] Lustre: server umount lustre-MDT0000 complete [ 6431.716867] LustreError: 135992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.735255] LustreError: 135992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6434.374888] LustreError: 135975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158726 with bad export cookie 4851557107003667548 [ 6434.383422] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6434.842571] Lustre: server umount lustre-MDT0001 complete [ 6446.137739] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 6449.175403] Key type lgssc unregistered [ 6449.428970] LNet: 147107:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6449.438597] LNetError: 147107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6450.472269] LNet: Removed LNI 192.168.206.146@tcp [ 6451.747615] Key type .llcrypt unregistered [ 6451.751507] Key type ._llcrypt unregistered [ 6475.872889] Key type ._llcrypt registered [ 6475.876080] Key type .llcrypt registered [ 6475.994935] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 6492.386879] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 6493.560075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6493.593989] alg: No test for adler32 (adler32-zlib) [ 6494.803749] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6494.999557] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 6496.744272] Key type lgssc registered [ 6497.889492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6532.590757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6532.609420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6533.843445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6533.871742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6533.940409] Lustre: lustre-MDT0000: new disk, initializing [ 6534.033407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6534.093064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6538.755269] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6550.109038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6550.179227] Lustre: 150310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6550.246379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6550.249729] Lustre: Skipped 1 previous similar message [ 6550.355497] Lustre: lustre-MDT0001: new disk, initializing [ 6550.442305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6550.475962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6550.481830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6555.091822] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6564.456797] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6569.709404] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6576.398317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6576.647707] Lustre: lustre-OST0000: new disk, initializing [ 6576.652804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6576.659344] Lustre: 152278:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6576.757619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6582.360953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6582.378986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6582.536388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6583.804606] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6592.682578] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6598.448848] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_module obdecho/obdecho [ 6600.471036] LustreError: 153284:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6600.488579] LustreError: 153284:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6602.418538] LustreError: 153380:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6611.938404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6611.947140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.957733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6616.555681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6616.561940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.107944] Lustre: server umount lustre-MDT0000 complete [ 6621.813933] LustreError: 150302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158913 with bad export cookie 13384886470235448052 [ 6621.827686] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6622.130919] Lustre: server umount lustre-MDT0001 complete [ 6637.618029] Lustre: server umount lustre-OST0000 complete [ 6658.313281] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 6668.197509] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 6705.907737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.114391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6706.137725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6706.241764] Lustre: lustre-MDT0000: new disk, initializing [ 6706.300416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6706.316901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6710.063248] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6720.295951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6720.452817] Lustre: 157096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6720.570648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6720.575193] Lustre: Skipped 1 previous similar message [ 6720.674408] Lustre: lustre-MDT0001: new disk, initializing [ 6720.779497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6720.835236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6720.855359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6725.653820] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6734.424638] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6740.104807] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6746.611878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6746.878473] Lustre: lustre-OST0000: new disk, initializing [ 6746.883274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6746.889662] Lustre: 159064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6746.964553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6752.787903] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6755.360565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6755.376725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6755.493352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6762.205290] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6767.269039] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6767.427785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6772.261683] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6772.464626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6776.309754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6776.311932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6776.314966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6776.314973] Lustre: Skipped 2 previous similar messages [ 6776.317363] LustreError: Skipped 1 previous similar message [ 6776.327022] Lustre: Skipped 2 previous similar messages [ 6780.432630] Lustre: server umount lustre-OST0000 complete [ 6786.530941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.538408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6786.546722] Lustre: Skipped 1 previous similar message [ 6786.559796] Lustre: Skipped 1 previous similar message [ 6790.837815] Lustre: server umount lustre-MDT0000 complete [ 6791.653579] LustreError: 157907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6791.680816] LustreError: 157907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6795.195737] LustreError: 157090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159087 with bad export cookie 13384886470235449011 [ 6795.196815] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6795.211736] LustreError: 157090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6795.518271] Lustre: server umount lustre-MDT0001 complete [ 6799.368898] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:51:30 (1788159090) [ 6801.223404] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6804.174290] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:51:34 (1788159094) [ 6813.261911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6813.905170] LustreError: 161479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6813.998081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6819.070198] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6819.299134] LustreError: 161480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.395147] LustreError: 161479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6828.518142] LustreError: 161480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6828.640675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.903746] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6841.893424] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6847.501224] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6854.047897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6854.354955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6854.371362] Lustre: Skipped 1 previous similar message [ 6860.749225] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 6868.283331] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6879.016723] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.46@tcp (stopping) [ 6879.026550] Lustre: Skipped 1 previous similar message [ 6879.201924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.219691] Lustre: Skipped 1 previous similar message [ 6880.089031] Lustre: server umount lustre-MDT0000 complete [ 6884.119016] LustreError: 161475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6884.267532] LustreError: 161460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159176 with bad export cookie 13384886470235449893 [ 6884.275831] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6884.277619] LustreError: 161460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6889.264619] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.46@tcp (stopping) [ 6889.270411] Lustre: Skipped 4 previous similar messages [ 6890.814693] Lustre: server umount lustre-MDT0001 complete [ 6897.800744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6909.822497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.038483] LustreError: 165321:0:(llog.c:1655:llog_backup()) MGC192.168.206.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6910.047491] Lustre: 165321:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6914.018164] LustreError: 165326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.043330] LustreError: 165326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6949.857308] LustreError: 165328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6949.876357] LustreError: 165328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 6970.848356] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6970.862443] Lustre: 165321:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7011.843085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7016.214194] LustreError: 165328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7016.234882] LustreError: 165328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 7016.939849] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7017.130126] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7025.865102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7033.814913] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7034.859598] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7042.448276] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7044.250941] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7050.575195] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7052.234927] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7073.991642] Lustre: server umount lustre-MDT0000 complete [ 7075.816941] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7075.837222] Lustre: Skipped 3 previous similar messages [ 7075.841644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7077.757242] LustreError: 161459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159369 with bad export cookie 13384886470235456347 [ 7077.765306] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7077.770708] LustreError: 161459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7080.330841] Lustre: server umount lustre-MDT0001 complete [ 7094.852494] Lustre: server umount lustre-OST0000 complete [ 7116.197713] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 7125.326293] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7161.573396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.750868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7161.779811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7161.852324] Lustre: lustre-MDT0000: new disk, initializing [ 7161.926658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7161.937722] Lustre: Skipped 1 previous similar message [ 7161.951572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7166.330273] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7176.387035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7176.499465] Lustre: 170425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7176.543196] LustreError: 170436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7176.566039] LustreError: 170436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 7176.739354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7180.740906] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7188.438660] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7193.358236] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7199.129938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7199.342751] Lustre: lustre-OST0000: new disk, initializing [ 7199.345099] Lustre: Skipped 1 previous similar message [ 7199.347486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7199.350922] Lustre: Skipped 2 previous similar messages [ 7199.354645] Lustre: 172392:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7204.728429] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7206.955076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7206.966940] Lustre: Skipped 1 previous similar message [ 7206.969460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7207.011951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7212.070949] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7216.143925] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7216.304291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7220.559140] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7220.804792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7227.366245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7227.366753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.382088] Lustre: Skipped 1 previous similar message [ 7227.389497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7227.394051] Lustre: Skipped 2 previous similar messages [ 7228.847816] Lustre: server umount lustre-OST0000 complete [ 7238.344348] Lustre: server umount lustre-MDT0000 complete [ 7241.751844] LustreError: 171219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159533 with bad export cookie 13384886470235473182 [ 7241.753918] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7241.768777] LustreError: 171219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7242.180877] Lustre: server umount lustre-MDT0001 complete [ 7251.877504] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:59:02 (1788159542) [ 7258.197974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.632487] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7272.859358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7278.466960] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7286.418912] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7292.331565] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7299.748959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7307.064756] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7315.456268] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7329.878593] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:00:20 (1788159620) [ 7338.983916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7338.990252] LustreError: Skipped 1 previous similar message [ 7338.994237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.003336] Lustre: Skipped 3 previous similar messages [ 7339.007629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7339.012636] Lustre: Skipped 4 previous similar messages [ 7341.033644] Lustre: server umount lustre-OST0000 complete [ 7351.558601] Lustre: server umount lustre-MDT0000 complete [ 7355.615327] LustreError: 174782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159647 with bad export cookie 13384886470235474064 [ 7355.632166] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7355.640368] LustreError: 174782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7355.908385] Lustre: server umount lustre-MDT0001 complete [ 7367.962648] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 7370.797425] Key type lgssc unregistered [ 7371.058574] LNet: 178547:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7371.073273] LNetError: 178547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7372.138213] LNet: Removed LNI 192.168.206.146@tcp [ 7373.132136] Key type .llcrypt unregistered [ 7373.135478] Key type ._llcrypt unregistered [ 7393.376815] Key type ._llcrypt registered [ 7393.380258] Key type .llcrypt registered [ 7393.552517] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7394.765443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7395.027111] alg: No test for adler32 (adler32-zlib) [ 7396.312342] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7396.675765] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 7398.440276] Key type lgssc registered [ 7399.738752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7408.980548] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7409.008091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7424.608386] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7429.728611] LustreError: 179793:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.146@tcp: failed processing log, type 4: rc = -110 [ 7434.848449] LustreError: 179793:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7434.861354] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7434.865114] LustreError: MGC192.168.206.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7434.871902] LustreError: 179793:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7434.877413] LustreError: 179793:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -22 [ 7434.880174] LustreError: 179793:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 7434.922771] Lustre: server umount lustre-OST0000 complete [ 7434.925161] LustreError: 179793:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7444.152472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7445.639890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7449.677860] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7455.202826] LustreError: 180293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7455.234866] LustreError: 180293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7458.365855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7458.691743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7463.198475] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7470.600288] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7476.284671] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7482.150568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7482.347519] LustreError: 182110:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7482.362404] Lustre: 182110:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7482.562429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7487.789182] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7493.887179] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7496.162957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7496.170224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7496.185085] Lustre: Skipped 2 previous similar messages [ 7496.187928] Lustre: Skipped 1 previous similar message [ 7501.284696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7501.290975] Lustre: Skipped 3 previous similar messages [ 7501.982848] Lustre: server umount lustre-MDT0000 complete [ 7505.654650] LustreError: 180273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159797 with bad export cookie 16031179200268858702 [ 7505.662870] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7505.678940] LustreError: 180273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7505.995740] Lustre: server umount lustre-MDT0001 complete [ 7520.413655] Lustre: server umount lustre-OST0000 complete [ 7527.965984] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 7535.798481] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7567.736773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7568.052610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7568.071157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7568.130103] Lustre: lustre-MDT0000: new disk, initializing [ 7568.177416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7568.193310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7572.981529] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7583.269141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.367200] Lustre: 185630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7583.432741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7583.436775] Lustre: Skipped 1 previous similar message [ 7583.498783] Lustre: lustre-MDT0001: new disk, initializing [ 7583.586374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7583.620697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7583.627417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7587.395399] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7596.297370] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7601.911277] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7608.375071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7608.634553] Lustre: lustre-OST0000: new disk, initializing [ 7608.640386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7608.646662] Lustre: 187598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7608.767630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7614.012419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7614.019681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7614.150289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7614.200116] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7622.616302] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7627.500592] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7627.726902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7632.738570] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7633.045451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7634.913197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7634.926951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7634.940944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7634.948342] Lustre: Skipped 1 previous similar message [ 7640.035403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7640.039740] Lustre: Skipped 1 previous similar message [ 7645.166837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7645.173021] Lustre: Skipped 1 previous similar message [ 7649.249216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7649.427456] Lustre: server umount lustre-OST0000 complete [ 7655.393488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.411534] Lustre: Skipped 1 previous similar message [ 7655.416445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7655.420893] Lustre: Skipped 1 previous similar message [ 7667.169365] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7667.460566] Lustre: server umount lustre-MDT0000 complete [ 7670.756175] LustreError: 185642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7670.779300] LustreError: 185642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7671.057634] LustreError: 185621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159963 with bad export cookie 16031179200268859423 [ 7671.066446] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7671.073994] LustreError: 185621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7671.419851] Lustre: server umount lustre-MDT0001 complete [ 7680.249820] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:06:11 (1788159971) [ 7695.873661] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 7697.964303] Key type lgssc unregistered [ 7698.216129] LNet: 190542:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7698.223897] LNetError: 190542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7698.238253] LNet: Removed LNI 192.168.206.146@tcp [ 7698.891536] Key type .llcrypt unregistered [ 7698.893248] Key type ._llcrypt unregistered [ 7715.872188] Key type ._llcrypt registered [ 7715.875386] Key type .llcrypt registered [ 7715.986167] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7717.162177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7717.249470] alg: No test for adler32 (adler32-zlib) [ 7718.366618] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7718.855067] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 7720.504299] Key type lgssc registered [ 7721.364598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7728.042681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7728.070855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.521845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7733.651334] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7739.876133] LustreError: 191760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7739.906496] LustreError: 191760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7743.319362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7743.638236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7748.366347] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7757.897254] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7765.128205] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7773.055921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7773.233050] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7773.235799] LustreError: 193626:0:(llog.c:1696:llog_backup()) MGC192.168.206.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7773.241847] Lustre: 193626:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7773.384709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7779.159625] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7787.601307] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7794.147757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7794.159456] Lustre: Skipped 2 previous similar messages [ 7794.159674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7794.169955] Lustre: Skipped 2 previous similar messages [ 7796.900121] Lustre: server umount lustre-MDT0000 complete [ 7799.271244] LustreError: 191755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7799.320666] LustreError: 191755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7801.452880] LustreError: 191741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160093 with bad export cookie 1866785625899135782 [ 7801.460439] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7801.472255] LustreError: 191741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7801.879637] Lustre: server umount lustre-MDT0001 complete [ 7815.382489] Lustre: server umount lustre-OST0000 complete [ 7825.816950] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 7837.153043] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 7877.389694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7877.745804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7877.780720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7877.872719] Lustre: lustre-MDT0000: new disk, initializing [ 7877.972311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7878.003162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7882.414840] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7894.039543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7894.113565] Lustre: 197151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7894.149644] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7894.151497] Lustre: Skipped 1 previous similar message [ 7894.154067] LustreError: 197934:0:(llog.c:1696:llog_backup()) MGC192.168.206.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7894.158661] LustreError: 197934:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7894.164031] Lustre: 197934:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7894.176418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7894.179589] Lustre: Skipped 1 previous similar message [ 7894.254864] Lustre: lustre-MDT0001: new disk, initializing [ 7894.373273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7894.409879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7894.419879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7900.447449] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7910.209022] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7915.565311] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7923.286230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7923.506653] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7923.511401] Lustre: Skipped 1 previous similar message [ 7923.513644] LustreError: 199096:0:(llog.c:1696:llog_backup()) MGC192.168.206.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7923.519982] LustreError: 199096:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7923.524872] Lustre: 199096:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7923.531996] Lustre: 199096:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7923.548001] Lustre: lustre-OST0000: new disk, initializing [ 7923.551542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7923.557944] Lustre: 199119:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7923.646718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7929.671749] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 7932.993284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7933.006372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7933.099314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7940.137048] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7945.710225] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7946.078861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7950.931896] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7951.183976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7953.894607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.897128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7953.897587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7953.911748] Lustre: Skipped 1 previous similar message [ 7959.011578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7959.023113] Lustre: Skipped 2 previous similar messages [ 7959.507268] Lustre: server umount lustre-OST0000 complete [ 7964.131613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.154330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7964.167086] Lustre: Skipped 1 previous similar message [ 7969.249944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7969.268823] Lustre: Skipped 1 previous similar message [ 7969.650748] Lustre: server umount lustre-MDT0000 complete [ 7973.208595] LustreError: 197143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160265 with bad export cookie 1866785625899136503 [ 7973.214861] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7973.241929] LustreError: 197143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7973.670601] Lustre: server umount lustre-MDT0001 complete [ 7982.879488] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:11:13 (1788160273) [ 7999.877744] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 8009.567684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.013049] LustreError: 202249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8010.055434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8014.308713] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8015.337457] LustreError: 202250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8020.450826] LustreError: 202249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8023.991313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8030.075521] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8033.271666] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8039.847193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8046.212484] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8053.524093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8053.666457] Lustre: lustre-OST0001: new disk, initializing [ 8053.669561] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8053.674738] Lustre: 204560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8053.749912] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8053.754489] Lustre: Skipped 2 previous similar messages [ 8058.240889] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8062.514889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8062.525223] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8062.602877] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8066.739682] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8071.688685] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8073.665581] Lustre: DEBUG MARKER: ADD OST3 [ 8079.030106] loop: module loaded [ 8079.118261] loop0: detected capacity change from 0 to 204800000 [ 8079.300135] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8079.312442] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8079.358130] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8085.234422] loop0: detected capacity change from 0 to 204800000 [ 8086.212593] Lustre: DEBUG MARKER: STOP OST3 [ 8090.782975] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8090.965369] Lustre: lustre-OST0003: new disk, initializing [ 8090.970484] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8090.978710] Lustre: 206549:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8092.254412] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8092.387134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8092.391991] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8092.442743] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8094.304084] LustreError: 206669:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8094.987977] Lustre: Failing over lustre-OST0003 [ 8095.128541] Lustre: server umount lustre-OST0003 complete [ 8097.571341] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8097.773492] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8097.776329] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8097.782198] Lustre: Skipped 1 previous similar message [ 8097.784235] LustreError: 203740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8097.823573] LustreError: Skipped 1 previous similar message [ 8105.380317] Lustre: DEBUG MARKER: START OST3 again [ 8111.082028] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8111.315312] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8112.683357] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8113.274191] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8113.276359] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8116.480525] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8137.143593] Lustre: Failing over lustre-OST0003 [ 8137.196253] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8137.198277] LustreError: 205091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8137.205663] Lustre: Skipped 2 previous similar messages [ 8137.212571] LustreError: 205091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8137.282980] Lustre: server umount lustre-OST0003 complete [ 8157.668825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8157.674765] Lustre: Skipped 2 previous similar messages [ 8157.675567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8157.685736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8160.870440] Lustre: server umount lustre-MDT0000 complete [ 8164.221652] LustreError: 203761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160456 with bad export cookie 1866785625899137385 [ 8164.227118] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8164.243819] LustreError: 203761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8169.974455] LustreError: 202246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8169.990434] LustreError: 202246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 8170.742907] Lustre: server umount lustre-MDT0001 complete [ 8174.376664] Lustre: server umount lustre-OST0000 complete [ 8177.840738] Lustre: server umount lustre-OST0001 complete [ 8184.983835] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 8193.797931] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 8230.245146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8230.430243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8230.459861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8230.547548] Lustre: lustre-MDT0000: new disk, initializing [ 8230.617802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8230.623378] Lustre: Skipped 2 previous similar messages [ 8230.639996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8234.680689] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8245.109823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8245.237255] Lustre: 212016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8245.527976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8250.269357] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8260.210422] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8266.347451] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8275.078906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8275.579463] Lustre: lustre-OST0000: new disk, initializing [ 8275.582429] Lustre: Skipped 1 previous similar message [ 8275.586961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8275.592751] Lustre: Skipped 2 previous similar messages [ 8275.598132] Lustre: 213984:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8277.196592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8277.203056] Lustre: Skipped 1 previous similar message [ 8277.206466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8277.394988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8282.638870] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8292.785811] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8297.538679] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8297.747791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8301.984826] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8302.265622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8308.193930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8308.197055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8308.208459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8308.221859] Lustre: Skipped 4 previous similar messages [ 8308.237307] Lustre: Skipped 8 previous similar messages [ 8310.297687] Lustre: server umount lustre-OST0000 complete [ 8317.409533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8317.422442] LustreError: Skipped 1 previous similar message [ 8320.390771] Lustre: server umount lustre-MDT0000 complete [ 8323.556117] LustreError: 212026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8323.576174] LustreError: 212026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8323.734415] LustreError: 212008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160615 with bad export cookie 1866785625899139681 [ 8323.740996] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8323.745967] LustreError: 212008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8324.074831] Lustre: server umount lustre-MDT0001 complete [ 8327.901593] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:16:58 (1788160618) [ 8345.208660] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 8354.612897] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 8391.141322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8391.311834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8391.336701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8391.411121] Lustre: lustre-MDT0000: new disk, initializing [ 8391.489368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8391.495416] Lustre: Skipped 2 previous similar messages [ 8391.509075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8394.624309] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8405.309393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8405.420130] Lustre: 218594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8405.665549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8409.879937] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8418.111935] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8423.085787] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8429.696889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8429.978125] Lustre: 220561:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8431.463411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8435.225338] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8443.362799] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8447.011518] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8447.187618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8450.757196] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8450.936259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8457.185761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8457.198137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.206125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8457.216640] Lustre: Skipped 3 previous similar messages [ 8457.227589] Lustre: Skipped 4 previous similar messages [ 8458.657795] Lustre: server umount lustre-OST0000 complete [ 8468.524449] Lustre: server umount lustre-MDT0000 complete [ 8472.560877] LustreError: 218605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8472.580763] LustreError: 218605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8472.668072] LustreError: 218587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160764 with bad export cookie 1866785625899140563 [ 8472.672361] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8472.683830] LustreError: 218587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8473.246436] Lustre: server umount lustre-MDT0001 complete [ 8481.947237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8486.383591] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8494.417299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8499.136588] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8505.218812] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8510.044522] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8515.676841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8521.716898] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8528.605201] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8539.618449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8539.635939] LustreError: Skipped 2 previous similar messages [ 8539.641863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8539.657013] Lustre: Skipped 2 previous similar messages [ 8539.663297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8539.674575] Lustre: Skipped 4 previous similar messages [ 8544.325415] Lustre: server umount lustre-OST0000 complete [ 8555.586050] Lustre: server umount lustre-MDT0000 complete [ 8559.853133] LustreError: 222758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160851 with bad export cookie 1866785625899141445 [ 8559.860109] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8559.860774] LustreError: 222758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8560.083367] Lustre: server umount lustre-MDT0001 complete [ 8667.123988] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 8670.038085] Key type lgssc unregistered [ 8670.407323] LNet: 226442:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8670.416832] LNetError: 226442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8670.440281] LNet: Removed LNI 192.168.206.146@tcp [ 8671.643397] Key type .llcrypt unregistered [ 8671.645638] Key type ._llcrypt unregistered [ 8697.239217] Key type ._llcrypt registered [ 8697.241992] Key type .llcrypt registered [ 8697.390155] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 8712.646500] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 8713.714234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8713.769869] alg: No test for adler32 (adler32-zlib) [ 8714.971394] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8715.195659] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 8716.873092] Key type lgssc registered [ 8717.957377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8756.113603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8756.127716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8757.363343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8757.410754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8757.508364] Lustre: lustre-MDT0000: new disk, initializing [ 8757.588130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8757.615877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8761.846596] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8770.974508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8771.124285] Lustre: 229797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8771.202456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8771.206748] Lustre: Skipped 1 previous similar message [ 8771.295422] Lustre: lustre-MDT0001: new disk, initializing [ 8771.363438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8771.415109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8771.438735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8775.167517] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8784.436831] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8790.843979] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8798.039420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8798.341920] Lustre: lustre-OST0000: new disk, initializing [ 8798.353864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8798.365874] Lustre: 231766:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8798.477864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8803.945329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8803.964705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8804.044259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8805.054770] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8814.185163] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8818.894662] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8819.064882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8823.745590] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8824.129810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8829.925440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8829.929100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8829.940460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8829.954265] Lustre: Skipped 1 previous similar message [ 8829.968305] Lustre: Skipped 1 previous similar message [ 8831.849402] Lustre: server umount lustre-OST0000 complete [ 8838.122433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8838.133554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8838.154142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8840.170387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8840.178990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8841.500668] Lustre: server umount lustre-MDT0000 complete [ 8845.303148] LustreError: 230593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161137 with bad export cookie 14832548745653945646 [ 8845.311807] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8845.313936] LustreError: 230593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8845.650413] Lustre: server umount lustre-MDT0001 complete [ 8864.415431] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 8874.569786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8875.205161] LustreError: 234700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8875.226157] LustreError: 234700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8875.273117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.171146] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8880.612587] LustreError: 234701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8885.735924] LustreError: 234700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8889.506325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8889.803365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8894.462837] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8897.324669] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8904.034151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8904.318147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8910.668197] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8919.686459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8919.883363] Lustre: lustre-OST0001: new disk, initializing [ 8919.886769] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8919.895473] Lustre: 237008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8920.745585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8920.761184] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8920.877581] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8926.165670] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 8935.801874] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8941.430648] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8953.347286] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:27:24 (1788161244) [ 8956.385860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8956.394180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8956.420608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8956.427536] Lustre: Skipped 1 previous similar message [ 8961.615589] Lustre: server umount lustre-MDT0000 complete [ 8963.044719] LustreError: 234700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8963.065407] LustreError: 234700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8964.747843] LustreError: 234680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161256 with bad export cookie 14832548745653946528 [ 8964.755746] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8964.758911] LustreError: 234680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8964.997423] Lustre: server umount lustre-MDT0001 complete [ 8978.562672] Lustre: server umount lustre-OST0000 complete [ 8992.796718] Lustre: server umount lustre-OST0001 complete [ 9000.357575] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 9009.576421] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 9043.132924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9043.417358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9043.457604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9043.548021] Lustre: lustre-MDT0000: new disk, initializing [ 9043.651418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.658634] Lustre: Skipped 1 previous similar message [ 9043.669535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9048.532794] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9059.268955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9059.367852] Lustre: 240951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9059.421590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9059.425209] Lustre: Skipped 1 previous similar message [ 9059.493728] Lustre: lustre-MDT0001: new disk, initializing [ 9059.619621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9059.627988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9064.233988] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9074.659742] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9080.594251] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9087.461119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9087.757967] Lustre: lustre-OST0000: new disk, initializing [ 9087.770141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9087.778680] Lustre: 242922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9087.849301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9087.862198] Lustre: Skipped 1 previous similar message [ 9089.251130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9089.264645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9089.443198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9096.252560] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9108.128644] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9112.271559] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9112.597975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9117.947932] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9118.293556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9125.349970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9125.356644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9125.361899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9125.361906] Lustre: Skipped 4 previous similar messages [ 9125.376612] Lustre: Skipped 4 previous similar messages [ 9127.260527] Lustre: server umount lustre-OST0000 complete [ 9131.491077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9135.586740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9135.589352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9135.597679] Lustre: Skipped 1 previous similar message [ 9135.602885] Lustre: Skipped 3 previous similar messages [ 9137.706148] Lustre: server umount lustre-MDT0000 complete [ 9140.714265] LustreError: 240964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9140.727177] LustreError: 240964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9141.460618] LustreError: 240944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161433 with bad export cookie 14832548745653948663 [ 9141.461939] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9141.479476] LustreError: 240944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9141.897581] Lustre: server umount lustre-MDT0001 complete [ 9149.621520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9149.935184] LustreError: 245137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9154.059753] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9162.300128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9162.667591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9162.674769] Lustre: Skipped 1 previous similar message [ 9166.972974] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9173.892846] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9179.332455] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9186.111287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9192.639299] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9200.049716] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9348.579161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9348.585897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9348.586194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9348.602417] Lustre: Skipped 1 previous similar message [ 9361.376662] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9361.536659] Lustre: server umount lustre-OST0000 complete [ 9367.521156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9380.320259] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9380.449653] Lustre: server umount lustre-MDT0000 complete [ 9383.307962] LustreError: 247959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161675 with bad export cookie 14832548745653949545 [ 9383.311287] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9383.327613] LustreError: 247959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9383.683525] Lustre: server umount lustre-MDT0001 complete [ 9471.524226] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [ 9474.734673] Key type lgssc unregistered [ 9475.073816] LNet: 248746:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9475.084360] LNetError: 248746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9476.137634] LNet: Removed LNI 192.168.206.146@tcp [ 9477.071864] Key type .llcrypt unregistered [ 9477.076292] Key type ._llcrypt unregistered [ 9504.408163] Key type ._llcrypt registered [ 9504.411353] Key type .llcrypt registered [ 9504.551372] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 9522.103550] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 9523.015821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9523.023301] alg: No test for adler32 (adler32-zlib) [ 9524.221378] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9524.587026] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [ 9526.378675] Key type lgssc registered [ 9527.566727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9583.388126] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 9595.622331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9595.647953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9597.036762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9597.100992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9597.216259] Lustre: lustre-MDT0000: new disk, initializing [ 9597.309554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9597.365918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9603.105882] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9615.791517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9615.897868] Lustre: 252820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9615.985685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9615.991232] Lustre: Skipped 1 previous similar message [ 9616.085814] Lustre: lustre-MDT0001: new disk, initializing [ 9616.231110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9616.284275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9616.292485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9621.044865] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9626.216929] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9632.983991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9633.262641] Lustre: lustre-OST0000: new disk, initializing [ 9633.266963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9633.275902] Lustre: 254451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9633.418579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9635.036951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9635.044727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9635.161358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9640.452434] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9650.821619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9651.004055] Lustre: lustre-OST0001: new disk, initializing [ 9651.011376] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9651.015989] Lustre: 255323:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9651.118506] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9656.415741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9656.426888] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9656.546543] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9657.210325] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9669.930894] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9680.396876] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9693.183429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9693.197638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9693.230568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9697.775846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9697.778971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9697.807881] Lustre: Skipped 1 previous similar message [ 9697.845669] Lustre: Skipped 3 previous similar messages [ 9699.533708] Lustre: server umount lustre-MDT0000 complete [ 9703.687129] LustreError: 253614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161995 with bad export cookie 10769622895334247143 [ 9703.697212] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9703.714939] LustreError: 253614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9704.138553] Lustre: server umount lustre-MDT0001 complete [ 9718.499921] Lustre: server umount lustre-OST0000 complete [ 9731.233916] Lustre: server umount lustre-OST0001 complete [ 9745.274659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9755.604963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9763.823120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9774.594662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9796.736691] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [ 9808.077724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9808.096977] Lustre: lustre-MDT0000: reset Object Index mappings [ 9808.472500] LustreError: 260334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.484585] LustreError: 260334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9808.540936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9813.002102] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9813.992477] LustreError: 260335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9818.103151] LustreError: 260334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9822.318233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.334099] Lustre: lustre-MDT0001: reset Object Index mappings [ 9822.669366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9827.236493] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9830.632573] Lustre: 261476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9837.756907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9844.669367] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9847.345090] LustreError: 261831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9852.396964] LustreError: 262316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9853.366964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9853.708451] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9853.714372] Lustre: Skipped 1 previous similar message [ 9854.795416] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9854.825900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9859.920112] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [ 9867.700450] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9871.906227] Lustre: 263348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9879.015710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9879.028769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.040121] Lustre: Skipped 1 previous similar message [ 9879.046129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9884.896212] Lustre: server umount lustre-MDT0000 complete [ 9885.666164] LustreError: 260334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9885.694830] LustreError: 260334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9889.260393] LustreError: 261077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162181 with bad export cookie 10769622895334253933 [ 9889.287844] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9889.288513] LustreError: 261077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9889.735931] Lustre: server umount lustre-MDT0001 complete [ 9904.752920] Lustre: server umount lustre-OST0000 complete [ 9906.144138] Lustre: 250324:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788162182/real 1788162182] req@ffff8f2925521f80 x1875023564506368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788162198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9906.161803] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9906.168374] Lustre: Skipped 3 previous similar messages [ 9908.996207] Lustre: server umount lustre-OST0001 complete [ 9931.384641] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:43:41 (1788162221) [ 9948.873817] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [ 9961.310730] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [10005.955691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10006.222215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10006.282936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10006.366938] Lustre: lustre-MDT0000: new disk, initializing [10006.567072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10006.606064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10012.552894] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10025.882613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10026.026228] Lustre: 267602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10026.105125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10026.109910] Lustre: Skipped 1 previous similar message [10026.175879] Lustre: lustre-MDT0001: new disk, initializing [10026.319441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10026.345850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10031.602934] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10041.173522] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10047.171445] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10054.362666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10054.558833] Lustre: lustre-OST0000: new disk, initializing [10054.561766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10054.565478] Lustre: 269575:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10056.273442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10056.285095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10056.433706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10061.358435] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10070.267339] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10074.632287] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10074.792851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10079.159919] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10079.319871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10082.274263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10082.275582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10082.277623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10082.277631] Lustre: Skipped 4 previous similar messages [10082.300946] Lustre: Skipped 1 previous similar message [10087.398142] LustreError: 269667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10087.417513] LustreError: 269667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10087.548754] Lustre: server umount lustre-OST0000 complete [10092.527823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10092.532527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10092.542523] Lustre: Skipped 1 previous similar message [10092.550867] Lustre: Skipped 2 previous similar messages [10097.365763] Lustre: server umount lustre-MDT0000 complete [10100.816503] LustreError: 267595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162392 with bad export cookie 10769622895334256026 [10100.817590] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10100.830790] LustreError: 267595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10101.274791] Lustre: server umount lustre-MDT0001 complete [10119.712585] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [10130.306644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10130.729232] LustreError: 272507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10130.758660] LustreError: 272507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10130.790503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10130.796356] Lustre: Skipped 2 previous similar messages [10135.261972] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10142.942744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10146.759675] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10149.163721] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10154.865725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10160.023970] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10167.136930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10167.328947] Lustre: lustre-OST0001: new disk, initializing [10167.333323] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10167.343867] Lustre: 274817:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10168.995135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10169.001893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10169.083642] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10172.668040] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10181.423468] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10184.672231] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10189.034680] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10189.077970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10194.772844] Lustre: server umount lustre-MDT0000 complete [10194.935460] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10194.951635] Lustre: Skipped 1 previous similar message [10194.952596] LustreError: 272504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10194.968657] LustreError: 272504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10198.021870] LustreError: 272489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162490 with bad export cookie 10769622895334256908 [10198.029103] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10198.033028] LustreError: 272489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10200.038696] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10200.044725] Lustre: Skipped 1 previous similar message [10200.428676] Lustre: server umount lustre-MDT0001 complete [10204.501378] Lustre: server umount lustre-OST0000 complete [10209.229753] Lustre: server umount lustre-OST0001 complete [10226.586617] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [10236.559444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10241.892196] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10251.192745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10256.635910] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10259.546101] Lustre: 279006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10264.804858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10265.190683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10265.204685] Lustre: Skipped 5 previous similar messages [10269.872517] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10277.457906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10282.874208] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10282.879082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10283.825808] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10291.017315] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10294.892655] Lustre: 280882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10300.251564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10300.276131] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10310.008652] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:50:00 (1788162600) [10328.545442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10328.552518] LustreError: Skipped 1 previous similar message [10328.561959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10328.571630] Lustre: Skipped 3 previous similar messages [10328.579309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10330.961790] Lustre: server umount lustre-MDT0000 complete [10334.187790] LustreError: 277861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10334.224831] LustreError: 277861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [10335.060184] LustreError: 277846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162627 with bad export cookie 10769622895334263222 [10335.078171] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10335.092659] LustreError: 277846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10341.589314] Lustre: server umount lustre-MDT0001 complete [10351.851425] Lustre: server umount lustre-OST0000 complete [10355.421511] Lustre: server umount lustre-OST0001 complete [10361.259371] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [10370.965297] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [10419.757201] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [10429.486207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10429.725855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10429.763252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10429.817354] Lustre: lustre-MDT0000: new disk, initializing [10429.883502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10434.841915] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10447.425201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10447.558421] Lustre: 285122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10448.084180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10453.096207] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10465.248357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10465.606774] Lustre: 286757:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10467.642620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10472.326559] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10482.619056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10482.794185] Lustre: 287629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10484.489081] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10484.496171] Lustre: Skipped 1 previous similar message [10484.581661] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10489.416048] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10498.442489] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10501.571505] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10501.577706] Lustre: Skipped 1 previous similar message [10514.620449] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:53:25 (1788162805) [10519.524049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10519.529412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10519.538583] Lustre: Skipped 5 previous similar messages [10519.541342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10519.544180] Lustre: Skipped 8 previous similar messages [10522.645808] Lustre: server umount lustre-MDT0000 complete [10525.976308] LustreError: 285116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162818 with bad export cookie 10769622895334271090 [10525.985869] LustreError: 285116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10525.989110] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10526.315448] Lustre: server umount lustre-MDT0001 complete [10539.105416] Lustre: server umount lustre-OST0000 complete [10552.213715] Lustre: server umount lustre-OST0001 complete [10560.866981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10561.468259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10561.473094] Lustre: Skipped 5 previous similar messages [10566.336594] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10574.269881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10579.565916] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10586.332252] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10592.181457] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10599.756761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10606.272906] LustreError: 291702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10606.300210] LustreError: 291702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [10607.253438] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10615.891971] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10635.269664] loop1: detected capacity change from 0 to 204800000 [10635.460146] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10635.470573] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10635.480606] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10641.020710] loop1: detected capacity change from 0 to 204800000 [10646.578371] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10646.679680] LustreError: 289861:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs22252: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10646.702208] LustreError: 289861:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs22252 [10646.713573] LustreError: 289861:0:(mgs_handler.c:517:mgs_target_reg()) fs22252-MDT0001: Error checking target index: rc = -13 [10646.730650] LustreError: fs22252-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10646.741397] LustreError: 293592:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -13 [10646.747351] LustreError: 293592:0:(tgt_mount.c:2206:server_put_super()) no obd fs22252-MDT0001 [10646.752089] LustreError: 293592:0:(tgt_mount.c:116:server_deregister_mount()) fs22252-MDT0001 not registered [10646.814554] Lustre: server umount fs22252-MDT0001 complete [10646.824425] LustreError: 293592:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10655.670155] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10655.748677] Lustre: Setting parameter fs22252-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22252-MDT0001 [10655.796252] Lustre: srv-fs22252-MDT0001: No data found on store. Initialize space. [10655.801907] Lustre: Skipped 4 previous similar messages [10655.879443] Lustre: fs22252-MDT0001: new disk, initializing [10655.884480] Lustre: Skipped 3 previous similar messages [10655.990329] Lustre: srv-fs22252-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10655.996131] Lustre: 294059:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22252-MDT0001-mdtlov: can't create new llog: rc = -115 [10656.004742] LustreError: 294059:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22252-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10661.047822] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10668.039121] Lustre: server umount fs22252-MDT0001 complete [10682.550761] loop1: detected capacity change from 0 to 204800000 [10686.444351] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10688.649294] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.265719] Lustre: 295627:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22252-MDT0001-mdtlov: can't create new llog: rc = -115 [10689.286678] LustreError: 295627:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22252-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10694.655156] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10699.780655] Lustre: server umount fs22252-MDT0001 complete [10712.035990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10712.038311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10712.049100] Lustre: Skipped 5 previous similar messages [10712.063349] Lustre: Skipped 6 previous similar messages [10715.461429] Lustre: server umount lustre-MDT0000 complete [10724.271617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10724.467638] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10729.689260] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10729.985444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10737.377324] loop1: detected capacity change from 0 to 204800000 [10744.124982] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10744.693316] Lustre: 298077:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22252-MDT0001-mdtlov: can't create new llog: rc = -115 [10744.708066] LustreError: 298077:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22252-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10748.755325] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10753.647572] Lustre: server umount fs22252-MDT0001 complete [10778.600308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10778.615830] LustreError: Skipped 1 previous similar message [10782.895412] Lustre: server umount lustre-OST0000 complete [10792.081945] Lustre: server umount lustre-MDT0000 complete [10795.606243] LustreError: 289854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163087 with bad export cookie 10769622895334273771 [10795.615284] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10795.624039] LustreError: 289854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10796.004041] Lustre: server umount lustre-MDT0001 complete [10810.419967] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [10814.191747] Key type lgssc unregistered [10814.580692] LNet: 300423:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10814.584940] LNetError: 300423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10814.609446] LNet: Removed LNI 192.168.206.146@tcp [10815.647167] Key type .llcrypt unregistered [10815.649953] Key type ._llcrypt unregistered [10834.975246] Key type ._llcrypt registered [10834.977841] Key type .llcrypt registered [10835.069496] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [10835.913528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10835.958412] alg: No test for adler32 (adler32-zlib) [10836.967664] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10837.225957] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [10838.928181] Key type lgssc registered [10840.024077] Lustre: Echo OBD driver; http://www.lustre.org/ [10847.091555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10847.110819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10848.654367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10852.854130] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10858.990057] LustreError: 301711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10859.006857] LustreError: 301711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10860.917995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.299440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10866.111490] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10874.013935] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10879.871769] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10886.666400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10887.030417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10893.115077] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10894.263267] LustreError: 303542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10899.428832] LustreError: 303540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10900.254134] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10902.882386] LustreError: 303542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10907.935783] LustreError: 303542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10907.971241] LustreError: 303542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10916.840144] LustreError: 303542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10916.858725] LustreError: 303542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10921.333802] loop1: detected capacity change from 0 to 204800000 [10921.483298] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10921.495317] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10921.505171] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10926.233982] loop1: detected capacity change from 0 to 204800000 [10931.962446] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10932.153834] Lustre: fs30360-OST0000: new disk, initializing [10932.158156] Lustre: srv-fs30360-OST0000: No data found on store. Initialize space. [10932.164288] Lustre: 305445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10932.266036] Lustre: fs30360-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10933.525475] LustreError: 304212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10933.540291] LustreError: 304212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10938.756276] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10946.858509] Lustre: server umount fs30360-OST0000 complete [10958.536317] loop1: detected capacity change from 0 to 204800000 [10963.600148] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10963.843478] Lustre: fs30360-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10968.048429] LustreError: 303542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10968.077532] LustreError: 303542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [10968.949351] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [10973.168104] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10976.797311] loop2: detected capacity change from 0 to 204800000 [10977.009035] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10977.018688] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10977.026186] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10982.606400] loop2: detected capacity change from 0 to 204800000 [10988.059913] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10988.162826] LustreError: fs30360-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10988.168846] LustreError: fs30360-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10988.177210] LustreError: 308496:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -13 [10988.182744] LustreError: 308496:0:(tgt_mount.c:2206:server_put_super()) no obd fs30360-OST0001 [10988.186552] LustreError: 308496:0:(tgt_mount.c:116:server_deregister_mount()) fs30360-OST0001 not registered [10988.218994] Lustre: server umount fs30360-OST0001 complete [10988.225367] LustreError: 308496:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10988.527803] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10998.419464] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10998.596244] Lustre: fs30360-OST0001: new disk, initializing [10998.604097] Lustre: srv-fs30360-OST0001: No data found on store. Initialize space. [10998.621858] Lustre: 309046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10998.732214] Lustre: fs30360-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11005.707625] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11014.567041] Lustre: server umount fs30360-OST0000 complete [11024.104727] Lustre: server umount fs30360-OST0001 complete [11034.627719] LustreError: 303540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11034.651197] LustreError: 303540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11038.172628] loop1: detected capacity change from 0 to 204800000 [11044.780291] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11045.138114] Lustre: fs30360-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11052.564191] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11059.041279] loop2: detected capacity change from 0 to 204800000 [11063.976837] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11069.847277] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11077.968622] Lustre: server umount fs30360-OST0000 complete [11087.117644] Lustre: server umount fs30360-OST0001 complete [11101.156810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11101.170386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11101.172387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11101.183765] Lustre: Skipped 1 previous similar message [11106.287338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11111.398815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11111.408135] Lustre: Skipped 2 previous similar messages [11113.952127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11114.156950] Lustre: server umount lustre-OST0000 complete [11117.537691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11117.546184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11117.560843] Lustre: Skipped 1 previous similar message [11117.568418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11121.635096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11121.637890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11121.659017] Lustre: Skipped 1 previous similar message [11123.720052] Lustre: server umount lustre-MDT0000 complete [11127.550467] LustreError: 301692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163419 with bad export cookie 12316290695291259353 [11127.556073] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11127.559697] LustreError: 301692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11127.859463] Lustre: server umount lustre-MDT0001 complete [11137.772513] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [11140.080791] Key type lgssc unregistered [11140.289780] LNet: 314971:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11140.298212] LNetError: 314971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11140.315413] LNet: Removed LNI 192.168.206.146@tcp [11141.087495] Key type .llcrypt unregistered [11141.091506] Key type ._llcrypt unregistered [11171.769467] Key type ._llcrypt registered [11171.770992] Key type .llcrypt registered [11171.992104] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [11191.007780] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [11192.029861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11192.160298] alg: No test for adler32 (adler32-zlib) [11193.258058] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11193.489682] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [11195.144135] Key type lgssc registered [11196.253764] Lustre: Echo OBD driver; http://www.lustre.org/ [11234.256148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11234.290071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11235.613990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11235.647244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11235.731762] Lustre: lustre-MDT0000: new disk, initializing [11235.832600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11235.862685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11240.399205] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11250.036674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11250.158378] Lustre: 318831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11250.224627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11250.228468] Lustre: Skipped 1 previous similar message [11250.337670] Lustre: lustre-MDT0001: new disk, initializing [11250.429588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11250.470480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11250.491755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11254.923597] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11264.025953] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11269.283441] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11276.479264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11276.831406] Lustre: lustre-OST0000: new disk, initializing [11276.836211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11276.846500] Lustre: 320800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11276.929452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11283.668402] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11285.104160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11285.115074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11285.234251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11292.679139] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11294.801510] Lustre: Mounted lustre-client [11295.707654] Lustre: Unmounted lustre-client [11304.932818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11304.952452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11310.063206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11310.082202] Lustre: Skipped 1 previous similar message [11310.895958] Lustre: server umount lustre-OST0000 complete [11315.170967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11315.174832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11315.192773] Lustre: Skipped 2 previous similar messages [11315.201201] Lustre: Skipped 1 previous similar message [11320.289555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11320.297672] Lustre: Skipped 2 previous similar messages [11321.224625] Lustre: server umount lustre-MDT0000 complete [11325.072753] LustreError: 318824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163617 with bad export cookie 8668553688683149186 [11325.084962] LustreError: 318824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11325.085813] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11325.409823] Lustre: server umount lustre-MDT0001 complete [11336.412458] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [11339.159521] Key type lgssc unregistered [11349.152425] LNet: 1 local NIs in recovery (showing 1): 192.168.206.146@tcp [11349.664513] LNet: 322962:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11349.683451] LNetError: 322962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11425.519170] LNet: Removed LNI 192.168.206.146@tcp [11426.330167] Key type .llcrypt unregistered [11426.333395] Key type ._llcrypt unregistered [11443.870309] Key type ._llcrypt registered [11443.873095] Key type .llcrypt registered [11443.965607] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [11458.650415] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [11459.957726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11460.183861] alg: No test for adler32 (adler32-zlib) [11461.346243] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11461.610879] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [11463.305091] Key type lgssc registered [11464.502528] Lustre: Echo OBD driver; http://www.lustre.org/ [11491.878246] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:09:42 (1788163782) [11498.582300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11498.602269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11499.897480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11499.960899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11500.108463] Lustre: lustre-MDT0000: new disk, initializing [11500.207445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11500.239170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11504.913110] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11515.949194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11516.070143] Lustre: 326188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11516.133935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11516.137630] Lustre: Skipped 1 previous similar message [11516.220246] Lustre: lustre-MDT0001: new disk, initializing [11516.316781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11516.381637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11516.395346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11520.594848] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11530.874883] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11537.218235] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11543.325658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11543.524336] Lustre: lustre-OST0000: new disk, initializing [11543.528636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11543.535736] Lustre: 328154:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11543.619477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11549.293471] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11551.819519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11551.829625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11551.919960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11558.684611] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11572.716297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11572.720353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11572.731783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11577.829617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11577.830655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11577.850515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11579.158545] Lustre: server umount lustre-OST0000 complete [11582.946162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11582.959861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11582.975545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11582.976267] Lustre: Skipped 1 previous similar message [11582.984956] Lustre: Skipped 2 previous similar messages [11588.067826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11588.076290] Lustre: Skipped 1 previous similar message [11593.187915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11593.199168] Lustre: Skipped 1 previous similar message [11597.292557] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11597.502468] Lustre: server umount lustre-MDT0000 complete [11600.634173] LustreError: 326180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163892 with bad export cookie 3717496158786991827 [11600.636622] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11600.643872] LustreError: 326180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11600.929840] Lustre: server umount lustre-MDT0001 complete [11610.605439] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [11613.276875] Key type lgssc unregistered [11613.489994] LNet: 330063:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11613.500380] LNetError: 330063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11613.523064] LNet: Removed LNI 192.168.206.146@tcp [11614.213212] Key type .llcrypt unregistered [11614.215394] Key type ._llcrypt unregistered [11634.820465] Key type ._llcrypt registered [11634.823761] Key type .llcrypt registered [11634.945422] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [11635.942401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11636.046883] alg: No test for adler32 (adler32-zlib) [11637.091807] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11637.290550] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [11638.936176] Key type lgssc registered [11640.172253] Lustre: Echo OBD driver; http://www.lustre.org/ [11647.557563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11647.594175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11649.034185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11653.799243] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11659.236421] LustreError: 331340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11659.252614] LustreError: 331340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11662.515915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11662.961088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11667.718378] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11674.978298] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11679.436828] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11684.860341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11685.103415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11690.193014] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11696.864550] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11706.291202] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11707.661761] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11713.850436] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.46@tcp (stopping) [11714.019517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11714.026102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11715.043693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11715.049646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11715.051244] Lustre: Skipped 1 previous similar message [11715.063044] Lustre: Skipped 1 previous similar message [11717.771688] Lustre: server umount lustre-MDT0000 complete [11718.937400] LustreError: 332357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11720.166660] LustreError: 331340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11720.187880] LustreError: 331340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11720.825058] LustreError: 331319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164012 with bad export cookie 7646032478930645405 [11720.834724] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11720.839646] LustreError: 331319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11724.053513] LustreError: 331334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11724.081719] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.46@tcp (stopping) [11724.085374] Lustre: Skipped 2 previous similar messages [11725.295549] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11727.108093] Lustre: server umount lustre-MDT0001 complete [11729.541524] Lustre: server umount lustre-OST0000 complete [11735.578850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11738.933490] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11741.315232] Lustre: 335516:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.146@tcp [11743.125428] Lustre: server umount MGS complete [11749.919910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.258157] LustreError: 336087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11750.270559] LustreError: 336087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11750.309216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11753.643400] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11760.426745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11760.611983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11760.622523] Lustre: Skipped 1 previous similar message [11760.741697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11765.494392] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11772.990676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11773.296121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11777.942478] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11783.084809] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11784.247928] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11788.771837] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11790.278607] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11795.939752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11795.947942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11795.961063] Lustre: Skipped 1 previous similar message [11795.966713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11795.974693] Lustre: Skipped 1 previous similar message [11801.066234] LustreError: 337527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.085842] LustreError: 337527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11801.134460] Lustre: server umount lustre-OST0000 complete [11806.177988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11806.180992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11806.189602] Lustre: Skipped 1 previous similar message [11810.431626] Lustre: server umount lustre-MDT0000 complete [11813.434378] LustreError: 336825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164105 with bad export cookie 7646032478930646252 [11813.439899] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11813.443248] LustreError: 336825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11813.692533] Lustre: server umount lustre-MDT0001 complete [11821.978149] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [11824.087909] Key type lgssc unregistered [11824.281290] LNet: 339523:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11824.291815] LNetError: 339523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11824.304429] LNet: Removed LNI 192.168.206.146@tcp [11824.940177] Key type .llcrypt unregistered [11824.942878] Key type ._llcrypt unregistered [11837.526793] Key type ._llcrypt registered [11837.528097] Key type .llcrypt registered [11837.602756] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [11849.463503] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [11850.240016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11850.254152] alg: No test for adler32 (adler32-zlib) [11851.205599] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11851.355925] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [11853.010402] Key type lgssc registered [11853.861165] Lustre: Echo OBD driver; http://www.lustre.org/ [11880.398035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11880.437989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11881.624422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11881.644427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11881.711355] Lustre: lustre-MDT0000: new disk, initializing [11881.775994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11881.800095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11885.122753] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11893.075168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.221132] Lustre: 342660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11893.270442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11893.274263] Lustre: Skipped 1 previous similar message [11893.359194] Lustre: lustre-MDT0001: new disk, initializing [11893.442839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11893.483649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11893.491835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11896.880321] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11904.455534] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11909.214373] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11915.059394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11915.366661] Lustre: lustre-OST0000: new disk, initializing [11915.369770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11915.374848] Lustre: 344625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11915.476415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11920.624220] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [11922.501978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11922.515044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11922.606282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11928.817192] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11932.623906] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11932.825885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11936.974180] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11937.192923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11943.399948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11943.411822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11943.412218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11943.419816] Lustre: Skipped 1 previous similar message [11943.448342] LustreError: Skipped 1 previous similar message [11948.524508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11948.531491] Lustre: Skipped 1 previous similar message [11953.120077] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11953.265544] Lustre: server umount lustre-OST0000 complete [11958.755231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11958.758622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11958.768438] Lustre: Skipped 1 previous similar message [11958.778371] Lustre: Skipped 2 previous similar messages [11963.130271] Lustre: server umount lustre-MDT0000 complete [11966.605221] LustreError: 342651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164258 with bad export cookie 8458015009277017956 [11966.608207] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11966.615357] LustreError: 342651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11966.961031] Lustre: server umount lustre-MDT0001 complete [11974.685745] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:17:45 (1788164265) [11988.414237] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [11996.374542] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [12029.943763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12030.201510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12030.240470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12030.326231] Lustre: lustre-MDT0000: new disk, initializing [12030.415365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12030.429851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12035.307783] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [12044.672637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12044.750690] Lustre: 349383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12044.814148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12044.822021] Lustre: Skipped 1 previous similar message [12044.963379] Lustre: lustre-MDT0001: new disk, initializing [12045.092815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12045.138372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12045.145410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12049.305630] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [12057.653187] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12062.680399] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12069.099424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12069.402297] Lustre: lustre-OST0000: new disk, initializing [12069.408591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12069.415899] Lustre: 351351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12069.487534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12075.245405] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [12077.647042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12077.656148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12077.777331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12083.741580] Lustre: DEBUG MARKER: oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12113.894292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12113.896386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12113.898145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12113.899696] LustreError: Skipped 1 previous similar message [12113.926765] Lustre: Skipped 1 previous similar message [12116.048944] Lustre: server umount lustre-OST0000 complete [12119.013609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12119.023215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12119.034835] Lustre: Skipped 1 previous similar message [12119.047738] Lustre: Skipped 2 previous similar messages [12129.251414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12129.254796] Lustre: Skipped 3 previous similar messages [12133.344124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12133.550698] Lustre: server umount lustre-MDT0000 complete [12134.372476] LustreError: 352171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12134.387523] LustreError: 352171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12136.836397] LustreError: 349375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164428 with bad export cookie 8458015009277018838 [12136.838774] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12136.847723] LustreError: 349375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12137.071385] Lustre: server umount lustre-MDT0001 complete [12144.984392] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12147.129609] Key type lgssc unregistered [12147.391664] LNet: 353507:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12147.404938] LNetError: 353507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12147.418639] LNet: Removed LNI 192.168.206.146@tcp [12147.973157] Key type .llcrypt unregistered [12147.974347] Key type ._llcrypt unregistered [12227.190188] Key type ._llcrypt registered [12227.196637] Key type .llcrypt registered [12227.317255] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12227.569263] Key type .llcrypt unregistered [12227.573033] Key type ._llcrypt unregistered [12294.848401] Key type ._llcrypt registered [12294.856680] Key type .llcrypt registered [12294.965923] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12295.223894] Key type .llcrypt unregistered [12295.227149] Key type ._llcrypt unregistered [12319.851554] Key type ._llcrypt registered [12319.856032] Key type .llcrypt registered [12319.943843] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12320.190629] Key type .llcrypt unregistered [12320.192177] Key type ._llcrypt unregistered [12347.186734] Key type ._llcrypt registered [12347.188394] Key type .llcrypt registered [12347.276924] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12347.561546] Key type .llcrypt unregistered [12347.565793] Key type ._llcrypt unregistered [12415.820536] Key type ._llcrypt registered [12415.825369] Key type .llcrypt registered [12415.932675] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12416.249409] Key type .llcrypt unregistered [12416.252351] Key type ._llcrypt unregistered [12444.195605] Key type ._llcrypt registered [12444.203060] Key type .llcrypt registered [12444.347794] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12444.655422] Key type .llcrypt unregistered [12444.659095] Key type ._llcrypt unregistered [12470.788230] Key type ._llcrypt registered [12470.790531] Key type .llcrypt registered [12470.908403] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12471.190651] Key type .llcrypt unregistered [12471.193172] Key type ._llcrypt unregistered [12501.104200] Key type ._llcrypt registered [12501.105867] Key type .llcrypt registered [12501.252901] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12501.571675] Key type .llcrypt unregistered [12501.573620] Key type ._llcrypt unregistered [12530.449304] Key type ._llcrypt registered [12530.454792] Key type .llcrypt registered [12530.577366] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12530.979312] Key type .llcrypt unregistered [12530.983340] Key type ._llcrypt unregistered [12560.612202] Key type ._llcrypt registered [12560.614734] Key type .llcrypt registered [12560.741413] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12560.972349] Key type .llcrypt unregistered [12560.974593] Key type ._llcrypt unregistered [12631.015693] Key type ._llcrypt registered [12631.024280] Key type .llcrypt registered [12631.227228] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12631.619392] Key type .llcrypt unregistered [12631.624198] Key type ._llcrypt unregistered [12663.443242] Key type ._llcrypt registered [12663.445481] Key type .llcrypt registered [12663.583867] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12663.910316] Key type .llcrypt unregistered [12663.912029] Key type ._llcrypt unregistered [12693.741578] Key type ._llcrypt registered [12693.745934] Key type .llcrypt registered [12693.869777] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [12694.119371] Key type .llcrypt unregistered [12694.126189] Key type ._llcrypt unregistered [12748.538455] Key type ._llcrypt registered [12748.540104] Key type .llcrypt registered [12748.605183] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [12763.567505] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [12764.468428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12764.546304] alg: No test for adler32 (adler32-zlib) [12765.674927] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12765.896995] LNet: Added LNI 192.168.206.146@tcp [8/256/0/180] [12767.552154] Key type lgssc registered [12768.509624] Lustre: Echo OBD driver; http://www.lustre.org/ [12801.520360] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12801.539462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12802.830518] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12802.849393] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12802.914378] Lustre: tst250fs-MDT0000: new disk, initializing [12803.064215] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12803.104756] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12808.323942] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [12818.351537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12818.488392] Lustre: 373656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12818.550534] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12818.561188] Lustre: Skipped 1 previous similar message [12818.693487] Lustre: tst250fs-MDT0001: new disk, initializing [12818.835841] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12818.899430] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12818.920422] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12823.043712] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [12827.757906] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12833.901945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12834.151591] Lustre: tst250fs-OST0000: new disk, initializing [12834.158555] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12834.164552] Lustre: 375287:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12834.243120] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12835.414886] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12835.423510] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12835.527138] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12839.210758] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [12851.161465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12851.423315] Lustre: tst250fs-OST0001: new disk, initializing [12851.427043] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12851.435301] Lustre: 376159:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12851.546161] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12859.569648] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_default_debug -1 all [12861.043802] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12861.048715] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12861.226827] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12865.904993] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12868.033861] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [12868.040520] Lustre: Skipped 1 previous similar message [12887.016935] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12887.026184] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12887.030819] Lustre: Skipped 2 previous similar messages [12887.047765] Lustre: Skipped 3 previous similar messages [12892.136925] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12892.144131] Lustre: Skipped 2 previous similar messages [12897.254083] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12897.263961] Lustre: Skipped 4 previous similar messages [12900.320330] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12902.554831] Lustre: server umount tst250fs-MDT0000 complete [12906.067490] LustreError: 373648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165198 with bad export cookie 14636268952675854354 [12906.075241] LustreError: MGC192.168.206.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12906.082850] LustreError: 373648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12906.551465] Lustre: server umount tst250fs-MDT0001 complete [12920.753243] Lustre: server umount tst250fs-OST0000 complete [12923.680249] Lustre: 371881:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788165199/real 1788165199] req@ffff8f2808ae0380 x1875026962880128/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788165215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12923.718974] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12923.737026] Lustre: Skipped 1 previous similar message [12924.899553] Lustre: server umount tst250fs-OST0001 complete [12945.733456] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [12953.677752] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [12981.777611] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:34:33 (1788165273) [12983.054447] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13002.208377] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing set_hostid [13013.242759] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing load_modules_local [13046.656308] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12724 sec ========= 04:35:37 (1788165337) [13048.285931] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:35:39 (1788165339) === [13052.019340] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:35:42 (1788165342) === [13087.345662] Lustre: DEBUG MARKER: oleg646-server.virtnet: executing unload_modules_local [13089.735345] Key type lgssc unregistered [13089.947374] LNet: 385308:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13089.962103] LNetError: 385308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13089.985183] LNet: Removed LNI 192.168.206.146@tcp [13090.724151] Key type .llcrypt unregistered [13090.727056] Key type ._llcrypt unregistered