[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 542257311 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003174] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008788] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011154] LSM: Security Framework initializing [ 0.012055] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014079] *** VALIDATE selinux *** [ 0.027555] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.033389] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.034236] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035182] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037135] *** VALIDATE tmpfs *** [ 0.040026] *** VALIDATE proc *** [ 0.042292] *** VALIDATE cgroup *** [ 0.043014] *** VALIDATE cgroup2 *** [ 0.044304] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.045291] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.046018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.047060] Spectre V2 : User space: Vulnerable [ 0.048024] Speculative Store Bypass: Vulnerable [ 0.052139] debug: unmapping init [mem 0xffffffffa8c59000-0xffffffffa8c60fff] [ 0.055379] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.056842] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.057029] ... version: 2 [ 0.058012] ... bit width: 48 [ 0.059011] ... generic registers: 4 [ 0.060013] ... value mask: 0000ffffffffffff [ 0.061015] ... max period: 00007fffffffffff [ 0.062018] ... fixed-purpose events: 3 [ 0.063015] ... event mask: 000000070000000f [ 0.064329] rcu: Hierarchical SRCU implementation. [ 0.066682] smp: Bringing up secondary CPUs ... [ 0.067986] x86: Booting SMP configuration: [ 0.068026] .... node #0, CPUs: #1 #2 #3 [ 0.078532] smp: Brought up 1 node, 4 CPUs [ 0.080093] smpboot: Max logical packages: 1 [ 0.081017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.124749] node 0 deferred pages initialised in 41ms [ 0.128205] devtmpfs: initialized [ 0.130200] x86/mm: Memory block size: 128MB [ 0.133202] gcov: version magic: 0x41383552 [ 0.135532] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143784] pinctrl core: initialized pinctrl subsystem [ 0.146151] [ 0.146996] ************************************************************* [ 0.149010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152012] ** ** [ 0.155013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158015] ** ** [ 0.160019] ** This means that this kernel is built to expose internal ** [ 0.163013] ** IOMMU data structures, which may compromise security on ** [ 0.166016] ** your system. ** [ 0.169018] ** ** [ 0.172011] ** If you see this message and you are not debugging the ** [ 0.174018] ** kernel, report this immediately to your vendor! ** [ 0.176011] ** ** [ 0.178010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181014] ************************************************************* [ 0.184759] NET: Registered protocol family 16 [ 0.186425] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195726] cpuidle: using governor menu [ 0.197133] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198356] PCI: Using configuration type 1 for base access [ 0.199152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.207147] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.209015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212245] cryptd: max_cpu_qlen set to 1000 [ 0.216242] ACPI: Added _OSI(Module Device) [ 0.217014] ACPI: Added _OSI(Processor Device) [ 0.217977] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223230] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.228456] ACPI: Interpreter enabled [ 0.230045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.231008] ACPI: Using IOAPIC for interrupt routing [ 0.232107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234386] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.244068] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.245083] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.247015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.250245] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.255621] acpiphp: Slot [2] registered [ 0.258269] acpiphp: Slot [5] registered [ 0.260166] acpiphp: Slot [6] registered [ 0.262187] acpiphp: Slot [7] registered [ 0.264180] acpiphp: Slot [8] registered [ 0.266164] acpiphp: Slot [9] registered [ 0.268171] acpiphp: Slot [10] registered [ 0.269106] acpiphp: Slot [3] registered [ 0.271080] acpiphp: Slot [4] registered [ 0.272099] acpiphp: Slot [11] registered [ 0.274145] acpiphp: Slot [12] registered [ 0.276146] acpiphp: Slot [13] registered [ 0.277138] acpiphp: Slot [14] registered [ 0.279371] acpiphp: Slot [15] registered [ 0.281324] acpiphp: Slot [16] registered [ 0.283139] acpiphp: Slot [17] registered [ 0.284202] acpiphp: Slot [18] registered [ 0.286202] acpiphp: Slot [19] registered [ 0.288251] acpiphp: Slot [20] registered [ 0.290169] acpiphp: Slot [21] registered [ 0.292110] acpiphp: Slot [22] registered [ 0.293201] acpiphp: Slot [23] registered [ 0.295108] acpiphp: Slot [24] registered [ 0.297128] acpiphp: Slot [25] registered [ 0.298138] acpiphp: Slot [26] registered [ 0.300109] acpiphp: Slot [27] registered [ 0.302096] acpiphp: Slot [28] registered [ 0.303153] acpiphp: Slot [29] registered [ 0.305208] acpiphp: Slot [30] registered [ 0.307164] acpiphp: Slot [31] registered [ 0.308116] PCI host bridge to bus 0000:00 [ 0.310024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.318025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.324047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.330793] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.334523] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.346000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.350055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.352031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.354024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.356117] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.361035] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365184] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.368053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372047] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.378018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.391015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.397016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405049] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415032] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.426031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.465020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.481831] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.488027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.497025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.533019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.551115] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.566018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.579025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.621020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.645963] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.687015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.697022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.723022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.737213] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.758026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.777023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.818038] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.850064] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.870015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.876022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.899025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.909000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.917465] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.921320] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.924400] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.927242] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.932119] iommu: Default domain type: Passthrough [ 0.934490] SCSI subsystem initialized [ 0.936172] ACPI: bus type USB registered [ 0.937103] usbcore: registered new interface driver usbfs [ 0.939086] usbcore: registered new interface driver hub [ 0.941122] usbcore: registered new device driver usb [ 0.942185] pps_core: LinuxPPS API ver. 1 registered [ 0.944024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.947058] PTP clock support registered [ 0.949162] EDAC MC: Ver: 3.0.0 [ 0.952084] PCI: Using ACPI for IRQ routing [ 0.954973] NetLabel: Initializing [ 0.957014] NetLabel: domain hash size = 128 [ 0.959011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.961091] NetLabel: unlabeled traffic allowed by default [ 0.964378] vgaarb: loaded [ 0.966325] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.968023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.977000] clocksource: Switched to clocksource kvm-clock [ 1.150495] VFS: Disk quotas dquot_6.6.0 [ 1.155736] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.162483] *** VALIDATE ramfs *** [ 1.165353] *** VALIDATE hugetlbfs *** [ 1.169293] pnp: PnP ACPI init [ 1.174211] pnp: PnP ACPI: found 6 devices [ 1.192131] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.195783] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.198139] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.200527] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.203166] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.205829] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.208924] NET: Registered protocol family 2 [ 1.211814] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.216816] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.220222] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.225360] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.228728] TCP: Hash tables configured (established 65536 bind 65536) [ 1.231482] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.234349] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.237209] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.240299] NET: Registered protocol family 1 [ 1.244312] RPC: Registered named UNIX socket transport module. [ 1.246752] RPC: Registered udp transport module. [ 1.248854] RPC: Registered tcp transport module. [ 1.250622] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.253032] NET: Registered protocol family 44 [ 1.254694] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.257023] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.259467] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.261943] PCI: CLS 0 bytes, default 64 [ 1.263820] Unpacking initramfs... [ 3.056602] debug: unmapping init [mem 0xffff8d02fcc54000-0xffff8d02fffbffff] [ 3.061174] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.064162] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.068048] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.597424] Initialise system trusted keyrings [ 3.599052] Key type blacklist registered [ 3.601287] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.611447] zbud: loaded [ 3.614924] *** VALIDATE nfs *** [ 3.616103] *** VALIDATE nfs4 *** [ 3.617658] pstore: using deflate compression [ 3.621827] Platform Keyring initialized [ 3.742039] NET: Registered protocol family 38 [ 3.743838] Key type asymmetric registered [ 3.745389] Asymmetric key parser 'x509' registered [ 3.747169] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.749917] io scheduler mq-deadline registered [ 3.751555] io scheduler kyber registered [ 3.753126] io scheduler bfq registered [ 3.755267] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.758607] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.761966] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.764639] ACPI: Power Button [PWRF] [ 3.770893] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.781691] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.802144] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.811936] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.835567] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.864359] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.895071] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.900559] Non-volatile memory driver v1.3 [ 3.902568] Linux agpgart interface v0.103 [ 3.940758] virtio_blk virtio1: [vda] 146680 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.944130] vda: detected capacity change from 0 to 75100160 [ 3.966709] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.969987] vdb: detected capacity change from 0 to 1073741824 [ 3.985601] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.988901] vdc: detected capacity change from 0 to 2621440000 [ 4.011495] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.014924] vdd: detected capacity change from 0 to 2621440000 [ 4.041649] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.044924] vde: detected capacity change from 0 to 4294967296 [ 4.064777] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.068401] vdf: detected capacity change from 0 to 4294967296 [ 4.078639] libphy: Fixed MDIO Bus: probed [ 4.085856] usbcore: registered new interface driver usbserial_generic [ 4.088625] usbserial: USB Serial support registered for generic [ 4.091495] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.096539] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.098930] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.101737] mousedev: PS/2 mouse device common for all mice [ 4.104692] rtc_cmos 00:05: RTC can wake from S4 [ 4.109434] rtc_cmos 00:05: registered as rtc0 [ 4.109487] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.110787] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.116096] intel_pstate: CPU model not supported [ 4.119408] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.124315] hid: raw HID events driver (C) Jiri Kosina [ 4.126571] usbcore: registered new interface driver usbhid [ 4.126577] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.132030] usbhid: USB HID core driver [ 4.133768] drop_monitor: Initializing network drop monitor service [ 4.136371] Initializing XFRM netlink socket [ 4.138475] NET: Registered protocol family 10 [ 4.142054] Segment Routing with IPv6 [ 4.143782] NET: Registered protocol family 17 [ 4.146025] mpls_gso: MPLS GSO support [ 4.152527] RAS: Correctable Errors collector initialized. [ 4.155216] AVX version of gcm_enc/dec engaged. [ 4.156848] AES CTR mode by8 optimization enabled [ 4.262243] sched_clock: Marking stable (4262219716, 0)->(5482189116, -1219969400) [ 4.265585] registered taskstats version 1 [ 4.267912] Loading compiled-in X.509 certificates [ 4.270442] zswap: loaded using pool lzo/zbud [ 4.300147] Key type big_key registered [ 4.313991] Key type encrypted registered [ 4.315526] ima: No TPM chip found, activating TPM-bypass! [ 4.317349] ima: Allocated hash algorithm: sha1 [ 4.318671] ima: No architecture policies found [ 4.320249] evm: Initialising EVM extended attributes: [ 4.321692] evm: security.selinux [ 4.322718] evm: security.ima [ 4.323875] evm: security.capability [ 4.325145] evm: HMAC attrs: 0x1 [ 4.327231] rtc_cmos 00:05: setting system clock to 2026-08-27 14:22:25 UTC (1787840545) [ 4.333508] debug: unmapping init [mem 0xffffffffa9c03000-0xffffffffa9dfffff] [ 4.336117] debug: unmapping init [mem 0xffffffffa8982000-0xffffffffa8c58fff] [ 4.345071] Write protecting the kernel read-only data: 28672k [ 4.348257] debug: unmapping init [mem 0xffffffffa7003000-0xffffffffa71fffff] [ 4.350587] debug: unmapping init [mem 0xffffffffa7914000-0xffffffffa79fffff] [ 4.385512] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.393562] systemd[1]: Detected virtualization kvm. [ 4.395659] systemd[1]: Detected architecture x86-64. [ 4.397428] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.427189] systemd[1]: No hostname configured. [ 4.429488] systemd[1]: Set hostname to . [ 4.434532] random: systemd: uninitialized urandom read (16 bytes read) [ 4.437938] systemd[1]: Initializing machine ID from random generator. [ 4.598248] random: systemd: uninitialized urandom read (16 bytes read) [ 4.601374] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.606707] random: systemd: uninitialized urandom read (16 bytes read) [ 4.609367] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.619908] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.442649] device-mapper: uevent: version 1.0.3 [ 5.445207] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.201506] virtio_net virtio0 ens2: renamed from eth0 [ 6.230289] random: fast init done [ 6.335483] scsi host0: ata_piix [ 6.371370] scsi host1: ata_piix [ 6.372777] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.374704] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.829579] random: crng init done [ 10.831891] random: 7 urandom warning(s) missed due to ratelimiting [ 10.912797] dracut-initqueue[577]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.075191] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.288551] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.556737] SELinux: Disabled at runtime. [ 13.618470] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.626890] systemd[1]: Detected virtualization kvm. [ 13.628510] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.196648] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.200424] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.205488] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.210232] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.213587] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.220715] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.229233] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ 14.497530] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 14.907848] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.270885] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.294306] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.482843] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.507460] EDAC sbridge: Ver: 1.1.2 [ 17.181119] Key type dns_resolver registered [ 17.499976] NFS: Registering the id_resolver key type [ 17.502124] Key type id_resolver registered [ 17.503784] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg111-server login: [ 116.597835] libcfs: loading out-of-tree module taints kernel. [ 116.616945] Key type ._llcrypt registered [ 116.618844] Key type .llcrypt registered [ 116.693500] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 133.701318] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 135.228469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 135.237992] alg: No test for adler32 (adler32-zlib) [ 136.767733] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 137.615061] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 139.407170] Key type lgssc registered [ 141.396123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 155.493765] hrtimer: interrupt took 5753334 ns [ 159.067373] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 199.804392] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 212.629628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 212.662294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.936162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 213.957738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 214.045323] Lustre: lustre-MDT0000: new disk, initializing [ 214.160420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.178935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 218.534425] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.312848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.401699] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.422974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.426598] Lustre: Skipped 1 previous similar message [ 230.490601] Lustre: lustre-MDT0001: new disk, initializing [ 230.528700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.555933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.572689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 234.197418] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.841632] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 249.210748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.595186] Lustre: lustre-OST0000: new disk, initializing [ 249.602143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 249.608312] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.684371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 256.789192] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.674179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 258.697125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 258.769148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 276.293732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 276.506962] Lustre: lustre-OST0001: new disk, initializing [ 276.511235] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 276.520445] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 276.700142] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 285.182508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 285.200199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 285.250575] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 285.412252] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 304.144305] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 311.033655] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 319.525541] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing check_logdir /tmp/testlogs/ [ 325.968268] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing yml_node [ 333.090396] Lustre: DEBUG MARKER: Client: 2.17.57.68 [ 335.942453] Lustre: DEBUG MARKER: MDS: 2.17.57.68 [ 338.534181] Lustre: DEBUG MARKER: OSS: 2.17.57.68 [ 340.597599] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:27:59 EDT 2026 [ 357.129944] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 358.660208] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 377.322194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.323279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 377.343573] Lustre: Skipped 1 previous similar message [ 377.360934] Lustre: Skipped 3 previous similar messages [ 381.099932] Lustre: server umount lustre-MDT0000 complete [ 386.297167] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840927 with bad export cookie 12117026092255903245 [ 386.304938] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 386.305689] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 387.552434] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 387.554099] LustreError: 10098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.561972] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 387.570449] Lustre: Skipped 3 previous similar messages [ 387.615670] LustreError: 10098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 391.587830] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.588688] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 391.622390] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 391.651147] Lustre: Skipped 2 previous similar messages [ 392.988320] Lustre: server umount lustre-MDT0001 complete [ 397.533618] Lustre: server umount lustre-OST0000 complete [ 402.056054] Lustre: server umount lustre-OST0001 complete [ 410.282904] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 420.216548] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 458.637915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.998230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 459.021384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 459.089804] Lustre: lustre-MDT0000: new disk, initializing [ 459.143508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.159883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 464.497433] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 478.774600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.026981] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 479.123407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 479.138482] Lustre: Skipped 1 previous similar message [ 479.220332] Lustre: lustre-MDT0001: new disk, initializing [ 479.296684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 479.362120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 479.375557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 483.740657] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 499.287876] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 509.235277] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 521.356247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 522.227144] Lustre: lustre-OST0000: new disk, initializing [ 522.232127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 522.240291] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 522.426265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.808976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 523.825857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 524.070819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 532.314765] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 545.477939] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 552.516907] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 552.962702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 560.080696] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 560.462211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 565.255351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 565.259560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.278098] LustreError: Skipped 1 previous similar message [ 565.297207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 565.383121] Lustre: Skipped 1 previous similar message [ 570.346050] LustreError: 17118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.367180] LustreError: 17118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 570.698508] Lustre: server umount lustre-OST0000 complete [ 576.995796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 577.006378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.079442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 577.102952] Lustre: Skipped 1 previous similar message [ 585.697202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 585.706320] Lustre: Skipped 3 previous similar messages [ 591.327122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 593.816502] Lustre: server umount lustre-MDT0000 complete [ 594.916983] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.967627] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 600.213585] LustreError: 15954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.257747] LustreError: 15954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 601.094382] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841142 with bad export cookie 12117026092255904778 [ 601.095706] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.101271] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 601.831214] Lustre: server umount lustre-MDT0001 complete [ 611.819866] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:32:29 (1787841149) [ 639.593038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.412355] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.547750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 645.184665] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 655.404360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.188949] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 668.650726] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 673.946734] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 676.642659] Lustre: Failing over lustre-MDT0000 [ 677.043086] Lustre: server umount lustre-MDT0000 complete [ 681.440638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 681.442376] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 681.462322] Lustre: Skipped 2 previous similar messages [ 681.483279] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 697.411364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.510902] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 697.987694] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 702.051968] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 702.082170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 702.099800] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 704.713213] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 714.982536] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 722.912687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 722.922491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.928500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 722.933807] Lustre: Skipped 1 previous similar message [ 722.947937] Lustre: Skipped 3 previous similar messages [ 725.365254] Lustre: server umount lustre-MDT0000 complete [ 727.074751] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 727.092723] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 729.268786] LustreError: 19911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841270 with bad export cookie 12117026092255906192 [ 729.283110] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 729.982061] Lustre: server umount lustre-MDT0001 complete [ 739.546805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 740.120963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 740.127247] Lustre: Skipped 2 previous similar messages [ 744.997260] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 754.969435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.739876] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 769.213952] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 775.926983] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 778.779084] Lustre: Failing over lustre-MDT0000 [ 779.257740] Lustre: server umount lustre-MDT0000 complete [ 780.261803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 780.282364] Lustre: Skipped 1 previous similar message [ 791.524593] LustreError: 23265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 791.557546] LustreError: 23265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 796.641963] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841321/real 1787841321] req@ffff8d0381361a40 x1874686826977024/t0(0) o400->MGC192.168.201.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787841337 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 796.684136] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 799.782467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.865986] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa82853083b0d8d3a [ 805.880741] Lustre: MGC192.168.201.111@tcp: Connection restored to 0@lo (at 0@lo) [ 805.886894] Lustre: Skipped 1 previous similar message [ 806.293899] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 810.835260] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 811.506165] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 811.516651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 811.540405] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 820.996843] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 826.849197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 826.864287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.871136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 826.885363] Lustre: Skipped 1 previous similar message [ 826.930368] Lustre: Skipped 1 previous similar message [ 832.033858] Lustre: server umount lustre-MDT0000 complete [ 835.511983] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841376 with bad export cookie 12117026092255907130 [ 835.528599] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 835.529859] LustreError: 23252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 836.011125] Lustre: server umount lustre-MDT0001 complete [ 845.910199] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:36:24 (1787841384) [ 861.914821] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 870.754205] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 920.593916] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 938.548516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.976539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 939.029639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 939.148656] Lustre: lustre-MDT0000: new disk, initializing [ 939.280815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 939.283956] Lustre: Skipped 2 previous similar messages [ 939.352113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 945.406284] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 957.109685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.170878] Lustre: 29648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 957.391137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 963.169095] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 973.803556] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 978.963967] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 994.727862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 995.343223] Lustre: 31621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 997.040456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 997.141043] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1002.471538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1002.481945] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1002.484316] Lustre: Skipped 1 previous similar message [ 1002.487152] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 1005.318875] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1021.564391] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1027.554645] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1048.549578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1048.550334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.592606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.601875] Lustre: Skipped 3 previous similar messages [ 1052.715730] Lustre: server umount lustre-OST0000 complete [ 1063.906840] LustreError: 29655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.911871] Lustre: server umount lustre-MDT0000 complete [ 1063.919194] LustreError: 29655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 1067.776134] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841608 with bad export cookie 12117026092255907543 [ 1067.779929] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1067.783801] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1068.159594] Lustre: server umount lustre-MDT0001 complete [ 1081.649298] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1084.876670] Key type lgssc unregistered [ 1085.278380] LNet: 33604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.282213] LNetError: 33604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.296990] LNet: Removed LNI 192.168.201.111@tcp [ 1086.330325] Key type .llcrypt unregistered [ 1086.333745] Key type ._llcrypt unregistered [ 1115.076908] Key type ._llcrypt registered [ 1115.078965] Key type .llcrypt registered [ 1115.209958] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1117.034198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1117.146803] alg: No test for adler32 (adler32-zlib) [ 1118.389494] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1118.867484] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1120.647210] Key type lgssc registered [ 1122.074540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1131.652745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1131.670532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.083832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.196463] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1143.264338] LustreError: 35020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1143.276777] LustreError: 35020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1147.966208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.385937] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1148.636309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1153.182843] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1161.776134] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1170.372533] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1178.688303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1179.318245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.629784] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1193.688455] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1200.076956] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1219.432896] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:42:38 (1787841758) [ 1221.503597] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1223.548947] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1231.563428] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:42:50 (1787841770) [ 1241.186389] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:43:00 (1787841780) [ 1242.426524] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1250.916730] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1250.925467] Lustre: Skipped 2 previous similar messages [ 1252.952186] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:43:11 (1787841791) [ 1255.093544] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1255.101248] Lustre: Skipped 2 previous similar messages [ 1263.131755] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1263.140319] Lustre: Skipped 4 previous similar messages [ 1279.767857] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:43:38 (1787841818) [ 1283.552049] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1288.924204] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1299.993822] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:43:59 (1787841839) [ 1302.392633] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1302.395662] Lustre: Skipped 6 previous similar messages [ 1317.647629] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:44:16 (1787841856) [ 1336.288869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.294380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1336.295284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1336.315390] Lustre: Skipped 1 previous similar message [ 1341.419414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1341.432482] Lustre: Skipped 2 previous similar messages [ 1342.015580] Lustre: server umount lustre-OST0000 complete [ 1346.532705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.545883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.560158] Lustre: Skipped 1 previous similar message [ 1346.593135] Lustre: Skipped 1 previous similar message [ 1351.664378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1351.678899] Lustre: Skipped 1 previous similar message [ 1352.851496] Lustre: server umount lustre-MDT0000 complete [ 1356.768964] LustreError: 35043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.788975] LustreError: 35043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1357.244069] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841898 with bad export cookie 5426259462288526142 [ 1357.246169] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1357.258908] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1357.669160] Lustre: server umount lustre-MDT0001 complete [ 1372.829659] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 1376.424556] Key type lgssc unregistered [ 1376.770668] LNet: 42888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1376.780795] LNetError: 42888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.834055] LNet: Removed LNI 192.168.201.111@tcp [ 1378.939667] Key type .llcrypt unregistered [ 1378.941356] Key type ._llcrypt unregistered [ 1400.064149] Key type ._llcrypt registered [ 1400.067535] Key type .llcrypt registered [ 1400.210734] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 1417.212640] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 1418.688460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1418.747713] alg: No test for adler32 (adler32-zlib) [ 1419.929142] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 1420.285893] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 1422.019294] Key type lgssc registered [ 1423.159481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1464.838216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1464.879495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1466.195549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1466.232300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1466.306633] Lustre: lustre-MDT0000: new disk, initializing [ 1466.396300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1466.413664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1470.835416] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1482.140566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.280195] Lustre: 46050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1482.334191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1482.338443] Lustre: Skipped 1 previous similar message [ 1482.412843] Lustre: lustre-MDT0001: new disk, initializing [ 1482.507592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1482.557418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1482.574141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1486.796915] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1496.407735] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1501.743527] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1508.936893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1509.398705] Lustre: lustre-OST0000: new disk, initializing [ 1509.421709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1509.449521] Lustre: 48018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1509.595236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1515.965056] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1516.211811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1516.219257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1516.424527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1526.415893] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1531.572425] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1531.900533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1537.787398] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1537.984710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1542.124555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1542.128840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1542.159771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1542.198218] Lustre: Skipped 1 previous similar message [ 1547.239098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1547.248676] Lustre: Skipped 2 previous similar messages [ 1552.360039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1552.372911] Lustre: Skipped 1 previous similar message [ 1554.402372] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1554.558853] Lustre: server umount lustre-OST0000 complete [ 1559.528536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1559.544508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1559.570386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.595601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.835904] Lustre: server umount lustre-MDT0000 complete [ 1568.638779] LustreError: 46040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842109 with bad export cookie 17691035222398292311 [ 1568.651438] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.118440] Lustre: server umount lustre-MDT0001 complete [ 1580.269996] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:48:39 (1787842119) [ 1588.319836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.955332] LustreError: 50479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1588.990183] LustreError: 50479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1589.061462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.236508] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1594.336715] LustreError: 50480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1594.351690] LustreError: 50480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1599.461567] LustreError: 50479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1604.586049] LustreError: 50480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1605.704729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.131972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1611.119780] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1619.375664] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1626.846932] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1635.247771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1635.680326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.907449] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1654.238570] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1663.813415] LustreError: 53011:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1664.936435] Lustre: Modifying parameter general.timeout=1 in log params [ 1666.100621] Lustre: Modifying parameter general.timeout=2 in log params [ 1668.111808] Lustre: Modifying parameter general.timeout=4 in log params [ 1668.115610] Lustre: Skipped 1 previous similar message [ 1672.249877] Lustre: Modifying parameter general.timeout=8 in log params [ 1672.255397] Lustre: Skipped 3 previous similar messages [ 1680.260687] Lustre: Modifying parameter general.timeout=16 in log params [ 1680.273320] Lustre: Skipped 7 previous similar messages [ 1697.097213] Lustre: Modifying parameter general.timeout=33 in log params [ 1697.101475] Lustre: Skipped 16 previous similar messages [ 1729.891211] Lustre: Modifying parameter general.timeout=66 in log params [ 1729.898581] Lustre: Skipped 32 previous similar messages [ 1799.888618] Lustre: Disabling parameter general.timeout= in log params [ 1799.890740] Lustre: Skipped 63 previous similar messages [ 1803.007848] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:52:22 (1787842342) [ 1805.983186] Lustre: 59958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1807.994806] Lustre: 60056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1810.077529] Lustre: 60162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1815.479527] Lustre: 60466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1815.491686] Lustre: 60466:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1843.434769] Lustre: 61828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1843.465559] LustreError: 61828:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1848.019161] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:53:06 (1787842386) [ 1864.673047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.681748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1864.703626] Lustre: Skipped 2 previous similar messages [ 1864.722900] Lustre: Skipped 4 previous similar messages [ 1870.226894] Lustre: server umount lustre-MDT0000 complete [ 1874.391258] LustreError: 50460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842415 with bad export cookie 17691035222398293193 [ 1874.399203] LustreError: 50460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1874.410367] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1874.896866] Lustre: server umount lustre-MDT0001 complete [ 1890.780521] Lustre: server umount lustre-OST0000 complete [ 1921.998030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.373619] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1922.559812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1927.599497] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1939.604591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.668530] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1939.689929] Lustre: 64365:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1945.036180] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1953.334072] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1959.280305] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1967.508253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.721979] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1968.088082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.095790] Lustre: Skipped 1 previous similar message [ 1977.157618] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 1984.968380] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1986.442887] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1986.450074] Lustre: Skipped 23 previous similar messages [ 1986.472087] Lustre: 66932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1995.235279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.236214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1995.245383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1995.245391] Lustre: Skipped 3 previous similar messages [ 1995.252142] Lustre: Skipped 1 previous similar message [ 1995.255709] LustreError: Skipped 1 previous similar message [ 2000.069817] Lustre: server umount lustre-OST0000 complete [ 2005.476561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.498772] Lustre: Skipped 1 previous similar message [ 2010.875615] Lustre: server umount lustre-MDT0000 complete [ 2015.163621] LustreError: 64357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842556 with bad export cookie 17691035222398296287 [ 2015.174054] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.182213] LustreError: 64357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2015.584913] Lustre: server umount lustre-MDT0001 complete [ 2026.896595] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 2029.614408] Key type lgssc unregistered [ 2029.954919] LNet: 68144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.960628] LNetError: 68144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.018171] LNet: Removed LNI 192.168.201.111@tcp [ 2031.809664] Key type .llcrypt unregistered [ 2031.811136] Key type ._llcrypt unregistered [ 2055.497226] Key type ._llcrypt registered [ 2055.500791] Key type .llcrypt registered [ 2055.638522] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2056.854611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2057.047640] alg: No test for adler32 (adler32-zlib) [ 2058.162465] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2058.360567] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2060.023282] Key type lgssc registered [ 2061.530976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2069.803845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2069.834221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.428752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.238731] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2081.764598] LustreError: 69558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2081.796458] LustreError: 69558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2084.346761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.934959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2089.307768] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2097.998095] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2106.079661] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2112.962664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2113.508565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.387467] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2127.892336] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2136.388647] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2149.346622] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.359828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.374654] Lustre: Skipped 2 previous similar messages [ 2153.201806] Lustre: server umount lustre-MDT0000 complete [ 2154.479625] LustreError: 69559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.513970] LustreError: 69559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2157.105480] LustreError: 70299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842698 with bad export cookie 16887894152145162807 [ 2157.106376] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2157.124861] LustreError: 70299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2157.506834] Lustre: server umount lustre-MDT0001 complete [ 2172.407894] Lustre: server umount lustre-OST0000 complete [ 2200.448567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.897901] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2200.927927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2201.290444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.170747] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2213.691587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.738515] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2213.757473] Lustre: 74434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2214.094925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2218.597190] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2225.874814] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2232.325995] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2238.262332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2238.383243] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2238.692596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.646551] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2250.649853] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2252.150797] Lustre: 77010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2253.948512] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2258.913288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.923037] Lustre: Skipped 2 previous similar messages [ 2258.929414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2263.523041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2263.530875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.555021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2264.854284] Lustre: server umount lustre-OST0000 complete [ 2269.153573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.173163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.177214] Lustre: Skipped 3 previous similar messages [ 2274.274599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.288936] Lustre: Skipped 1 previous similar message [ 2274.868473] Lustre: server umount lustre-MDT0000 complete [ 2278.594584] LustreError: 77012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842819 with bad export cookie 16887894152145163654 [ 2278.606705] LustreError: 77012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2278.610955] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.156508] Lustre: server umount lustre-MDT0001 complete [ 2289.434204] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 2292.064168] Key type lgssc unregistered [ 2292.431364] LNet: 78225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2292.445014] LNetError: 78225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2292.480786] LNet: Removed LNI 192.168.201.111@tcp [ 2293.607839] Key type .llcrypt unregistered [ 2293.612281] Key type ._llcrypt unregistered [ 2317.444699] Key type ._llcrypt registered [ 2317.447397] Key type .llcrypt registered [ 2317.597575] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2318.438954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.572196] alg: No test for adler32 (adler32-zlib) [ 2319.753637] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2320.049220] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2321.751886] Key type lgssc registered [ 2323.237273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2331.892605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2331.914843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2333.433365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.547487] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2342.893155] LustreError: 79657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.912973] LustreError: 79657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2347.887862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2348.003159] LustreError: 79658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.557277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2353.079574] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2360.327664] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2366.656173] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2373.103095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2373.587642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2379.354602] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2387.857861] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2396.957729] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2397.745580] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2399.431219] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2399.436271] Lustre: Skipped 1 previous similar message [ 2401.987534] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2401.991848] Lustre: Skipped 2 previous similar messages [ 2407.065110] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2407.075179] Lustre: Skipped 4 previous similar messages [ 2416.008881] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2416.014213] Lustre: Skipped 7 previous similar messages [ 2432.122663] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2432.127400] Lustre: Skipped 14 previous similar messages [ 2464.715222] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2464.718942] Lustre: Skipped 24 previous similar messages [ 2514.921485] Lustre: 87770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2517.389468] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:04:16 (1787843056) [ 2518.835102] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2520.781313] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:04:19 (1787843059) [ 2526.179368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.181211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2526.198246] Lustre: Skipped 1 previous similar message [ 2526.206488] Lustre: Skipped 1 previous similar message [ 2531.300306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2531.324821] Lustre: Skipped 1 previous similar message [ 2531.644273] Lustre: server umount lustre-OST0000 complete [ 2536.417534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.427443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2536.436221] Lustre: Skipped 1 previous similar message [ 2536.461578] Lustre: Skipped 1 previous similar message [ 2541.545228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2541.550954] Lustre: Skipped 1 previous similar message [ 2541.648251] Lustre: server umount lustre-MDT0000 complete [ 2546.026169] LustreError: 85522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843087 with bad export cookie 11180908014811813602 [ 2546.035331] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.702496] Lustre: server umount lustre-MDT0001 complete [ 2559.562892] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 2562.698468] Key type lgssc unregistered [ 2563.020828] LNet: 89023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2563.026372] LNetError: 89023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2563.043978] LNet: Removed LNI 192.168.201.111@tcp [ 2563.882381] Key type .llcrypt unregistered [ 2563.884580] Key type ._llcrypt unregistered [ 2568.589680] Key type ._llcrypt registered [ 2568.591762] Key type .llcrypt registered [ 2568.799929] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2574.219963] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules [ 2575.238609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2575.473720] alg: No test for adler32 (adler32-zlib) [ 2576.717872] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2577.031792] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2578.714151] Key type lgssc registered [ 2580.067410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2581.605460] LustreError: 90226:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2599.832124] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2603.405438] LustreError: 90226:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2610.177656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2610.195891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.938539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.261759] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2621.412185] LustreError: 91251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.432747] LustreError: 91251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2626.532350] LustreError: 91250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.672242] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:06:05 (1787843165) [ 2631.655464] LustreError: 91251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.366915] LustreError: 91282:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2632.385470] LustreError: 91282:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2632.393803] LustreError: 91282:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2633.169593] Lustre: server umount lustre-MDT0000 complete [ 2650.553458] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 2653.694052] Key type lgssc unregistered [ 2654.074651] LNet: 92781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2654.081422] LNetError: 92781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2654.111323] LNet: Removed LNI 192.168.201.111@tcp [ 2655.324954] Key type .llcrypt unregistered [ 2655.329375] Key type ._llcrypt unregistered [ 2673.676479] Key type ._llcrypt registered [ 2673.682971] Key type .llcrypt registered [ 2673.783144] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2675.197694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2675.226948] alg: No test for adler32 (adler32-zlib) [ 2676.401159] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 2676.755344] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 2678.447371] Key type lgssc registered [ 2680.073493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2688.423164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2688.434800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.014081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2694.824386] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2700.263359] LustreError: 93999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.286376] LustreError: 93999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2704.856345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.859371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2712.047546] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2719.992772] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2728.249621] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2736.597665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.494279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.681643] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2751.065352] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2813.058902] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:09:11 (1787843351) [ 2814.969598] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2816.940317] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:09:16 (1787843356) [ 2821.603788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.613382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2821.630563] Lustre: Skipped 2 previous similar messages [ 2821.634902] Lustre: Skipped 1 previous similar message [ 2826.727383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.734662] Lustre: Skipped 3 previous similar messages [ 2827.532043] Lustre: server umount lustre-MDT0000 complete [ 2831.846235] LustreError: 93999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.872485] LustreError: 93999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2831.905929] LustreError: 93980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843373 with bad export cookie 13925146721257902970 [ 2831.924539] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2831.926397] LustreError: 93980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2832.531688] Lustre: server umount lustre-MDT0001 complete [ 2847.931465] Lustre: server umount lustre-OST0000 complete [ 2861.024782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.482243] LustreError: 98314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.822584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2867.001972] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2867.174700] LustreError: 98315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.273223] LustreError: 98314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.808532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2876.591687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2881.627917] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2889.440676] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2894.967918] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2907.607109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.961762] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2907.972290] LustreError: 98302:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2908.002579] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2908.012779] LustreError: 100400:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2908.019616] LustreError: 100400:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2908.027638] LustreError: 100400:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2908.084632] Lustre: server umount lustre-OST0000 complete [ 2908.086897] LustreError: 100400:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2917.972478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2918.218138] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2918.228351] LustreError: 98303:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2918.243709] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2918.259539] LustreError: 100862:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2918.277711] LustreError: 100862:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2918.298309] LustreError: 100862:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2918.443557] Lustre: server umount lustre-OST0000 complete [ 2918.447085] LustreError: 100862:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2928.663405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2928.813259] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2928.817354] Lustre: Found index 0 for lustre-OST0000, updating log [ 2928.821816] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2928.882716] Lustre: lustre-OST0000: new disk, initializing [ 2928.888503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2928.895285] Lustre: 101395:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2929.207505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.895679] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 2939.138361] Lustre: Failing over lustre-OST0000 [ 2939.336391] Lustre: server umount lustre-OST0000 complete [ 2946.405917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.411755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2946.425744] Lustre: Skipped 1 previous similar message [ 2946.451416] Lustre: Skipped 1 previous similar message [ 2951.651867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2951.657335] Lustre: Skipped 1 previous similar message [ 2953.771405] Lustre: server umount lustre-MDT0000 complete [ 2956.768628] LustreError: 98314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.794970] LustreError: 98314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2958.353360] LustreError: 98296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843499 with bad export cookie 13925146721257905714 [ 2958.359364] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2958.366325] LustreError: 98296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2958.903137] Lustre: server umount lustre-MDT0001 complete [ 2968.146795] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:11:47 (1787843507) [ 2984.934664] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 2996.101508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2996.622957] LustreError: 103741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.797880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.508732] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3010.064189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.357446] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3019.082549] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3027.206051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3028.304144] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3034.834519] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3035.431685] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3035.450555] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 3035.499965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3040.747265] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 3040.777649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3041.855303] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 3041.891410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 3041.907182] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 3041.908300] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3041.930644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 3044.880799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3045.001680] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3045.124467] Lustre: lustre-OST0001: new disk, initializing [ 3045.133904] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3045.139973] Lustre: 106058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3045.452573] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3045.462294] Lustre: Skipped 2 previous similar messages [ 3052.092414] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3054.769674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3054.775862] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3054.926433] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3060.839143] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3071.671919] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3077.608815] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3077.618508] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.651108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3080.678883] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.698073] Lustre: Skipped 1 previous similar message [ 3083.198907] Lustre: server umount lustre-MDT0001 complete [ 3085.798791] LustreError: 103741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.810151] LustreError: 103741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3093.086259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3093.202853] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3093.211279] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3093.301108] Lustre: 103731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3095.296349] LustreError: 107635:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3095.311366] LustreError: 107635:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3095.342034] LustreError: 103733:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3095.514460] LustreError: 107642:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.111@tcp: cfg command failed: rc = -17 [ 3095.530265] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3095.530265] [ 3098.871403] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3099.640202] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3109.510169] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 11:14:08 (1787843648) [ 3128.219850] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3144.007805] Lustre: 109670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3166.861058] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3170.418281] Lustre: 110806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3247.072042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3247.086744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.108599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.116709] Lustre: Skipped 4 previous similar messages [ 3252.186596] Lustre: server umount lustre-MDT0000 complete [ 3253.225190] LustreError: 103742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3253.235983] LustreError: 103742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3256.022781] LustreError: 103721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843797 with bad export cookie 13925146721257906645 [ 3256.026656] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3256.044406] LustreError: 103721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3256.458628] Lustre: server umount lustre-MDT0001 complete [ 3269.945603] Lustre: server umount lustre-OST0000 complete [ 3283.191214] Lustre: server umount lustre-OST0001 complete [ 3288.694988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3298.989549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3308.553664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3320.247399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3341.226636] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3350.583515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.602975] Lustre: lustre-MDT0000: reset Object Index mappings [ 3351.319852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.325236] Lustre: Skipped 1 previous similar message [ 3355.588910] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3364.479452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3364.523081] Lustre: lustre-MDT0001: reset Object Index mappings [ 3369.647385] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3372.218335] Lustre: 115921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3379.992075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3387.177732] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3388.915593] LustreError: 116275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3388.942928] LustreError: 116275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3388.943626] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3396.091332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3396.514250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3402.788189] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3402.810792] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3405.440815] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3415.775305] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3420.000456] Lustre: 117799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3466.446841] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:20:05 (1787844005) [ 3492.833704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3492.840230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.852500] Lustre: Skipped 3 previous similar messages [ 3492.856982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3492.861324] Lustre: Skipped 4 previous similar messages [ 3498.996448] Lustre: server umount lustre-MDT0000 complete [ 3502.859237] LustreError: 117801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844044 with bad export cookie 13925146721257953720 [ 3502.868195] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3502.873360] LustreError: 117801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3509.327395] Lustre: server umount lustre-MDT0001 complete [ 3514.212943] LustreError: 118562:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0xb7:0x0].0x0 (ffff8d024d262500) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3520.690530] Lustre: server umount lustre-OST0000 complete [ 3525.727806] LustreError: 118764:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000401:0x37:0x0].0x0 (ffff8d0348f31300) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3532.186503] Lustre: server umount lustre-OST0001 complete [ 3540.043723] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 3551.897181] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3586.474959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.855684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3586.892629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3586.982477] Lustre: lustre-MDT0000: new disk, initializing [ 3587.133525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3587.149363] Lustre: Skipped 3 previous similar messages [ 3587.174203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3592.178508] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3598.909643] Lustre: server umount lustre-MDT0000 complete [ 3607.808837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.089040] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3608.105414] Lustre: 122071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3608.120297] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3612.736633] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3617.970641] Lustre: server umount lustre-MDT0000 complete [ 3627.261301] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:22:46 (1787844166) [ 3628.720693] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3630.163461] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:22:49 (1787844169) [ 3647.612761] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 3658.529715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.862503] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3673.053481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3673.191322] Lustre: 124121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3673.254574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3673.262740] Lustre: Skipped 1 previous similar message [ 3673.342783] Lustre: lustre-MDT0001: new disk, initializing [ 3673.471067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3673.489386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3678.322220] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3683.507652] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3689.774126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3690.105925] Lustre: lustre-OST0000: new disk, initializing [ 3690.109822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3690.119479] Lustre: 125699:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3691.276725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3691.292595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3691.448750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3696.544912] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3708.265389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3708.573598] Lustre: lustre-OST0001: new disk, initializing [ 3708.581511] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3708.588962] Lustre: 126570:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3710.600656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3710.614699] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3710.715491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3715.084228] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 3725.220829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3728.985638] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4075.949414] Lustre: 132978:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4075.956260] Lustre: 132978:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4075.961234] Lustre: 132978:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4075.966951] Lustre: 132978:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4075.979605] Lustre: 132978:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4075.985577] Lustre: 132978:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4352.468836] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:34:50 (1787844890) [ 4369.893670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4369.904731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4369.927419] Lustre: Skipped 5 previous similar messages [ 4369.934451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4369.943738] Lustre: Skipped 8 previous similar messages [ 4375.754715] Lustre: server umount lustre-MDT0000 complete [ 4376.546741] LustreError: 126063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4376.581974] LustreError: 126063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 4380.960108] LustreError: 124112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844922 with bad export cookie 13925146721258020514 [ 4380.993910] LustreError: 124112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4380.996125] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4381.667744] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.692973] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4381.694874] Lustre: Skipped 4 previous similar messages [ 4381.708581] Lustre: Skipped 5 previous similar messages [ 4387.814384] Lustre: server umount lustre-MDT0001 complete [ 4398.660954] Lustre: server umount lustre-OST0000 complete [ 4409.472597] Lustre: server umount lustre-OST0001 complete [ 4416.968375] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 4429.168797] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 4474.905277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.278989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4475.313214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4475.437477] Lustre: lustre-MDT0000: new disk, initializing [ 4475.573364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4475.578169] Lustre: Skipped 5 previous similar messages [ 4475.592048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4480.739059] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4493.091084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4493.261581] Lustre: 136038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4493.345160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4493.359210] Lustre: Skipped 1 previous similar message [ 4493.521897] Lustre: lustre-MDT0001: new disk, initializing [ 4493.755732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4493.781190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4499.337838] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4512.325569] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4520.628791] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4531.469856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4531.850921] Lustre: lustre-OST0000: new disk, initializing [ 4531.861215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4531.874337] Lustre: 138008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4532.012197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4532.023397] Lustre: Skipped 1 previous similar message [ 4533.190415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4533.197225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4533.421896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4540.637923] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 4552.485691] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4564.718603] Lustre: lustre-MDD0000: changelog on [ 4568.702701] Lustre: lustre-MDD0001: changelog on [ 6268.125279] Lustre: lustre-MDD0001: changelog off [ 6270.315728] Lustre: 139818:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6271.263990] Lustre: lustre-MDD0000: changelog off [ 6279.158892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.159407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6279.160877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.160885] Lustre: Skipped 2 previous similar messages [ 6279.177549] Lustre: Skipped 1 previous similar message [ 6284.018243] Lustre: server umount lustre-OST0000 complete [ 6289.382423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.383838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6289.398246] Lustre: Skipped 1 previous similar message [ 6289.410719] Lustre: Skipped 2 previous similar messages [ 6294.500153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.618266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.624550] Lustre: Skipped 2 previous similar messages [ 6303.202402] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6303.638595] Lustre: server umount lustre-MDT0000 complete [ 6304.737311] LustreError: 136051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6304.770434] LustreError: 136051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6307.823028] LustreError: 136030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846848 with bad export cookie 13925146721258827817 [ 6307.835174] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6307.837864] LustreError: 136030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6308.567486] Lustre: server umount lustre-MDT0001 complete [ 6322.546548] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 6325.642320] Key type lgssc unregistered [ 6325.943885] LNet: 147126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6325.948952] LNetError: 147126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6327.034538] LNet: Removed LNI 192.168.201.111@tcp [ 6328.304166] Key type .llcrypt unregistered [ 6328.307973] Key type ._llcrypt unregistered [ 6351.364314] Key type ._llcrypt registered [ 6351.368769] Key type .llcrypt registered [ 6351.561383] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 6367.824377] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 6368.996066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6369.006447] alg: No test for adler32 (adler32-zlib) [ 6370.037370] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 6370.227572] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 6371.856640] Key type lgssc registered [ 6373.578301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6409.979561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6410.042808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.491466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6411.526793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6411.603317] Lustre: lustre-MDT0000: new disk, initializing [ 6411.710593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6411.737609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6417.039820] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6429.089050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.245864] Lustre: 150332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6429.304884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6429.309299] Lustre: Skipped 1 previous similar message [ 6429.395784] Lustre: lustre-MDT0001: new disk, initializing [ 6429.575654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6429.638777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6429.646594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6434.928271] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6444.103898] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6450.283177] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6456.637253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6456.888820] Lustre: lustre-OST0000: new disk, initializing [ 6456.891638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6456.895900] Lustre: 152300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6456.995435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6465.013829] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6465.619912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6465.628333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6465.865190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6475.478892] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6481.201598] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_module obdecho/obdecho [ 6483.562896] LustreError: 153304:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6483.572143] LustreError: 153304:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6485.743970] LustreError: 153401:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6496.225603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6496.233571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.251421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6496.741299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.763152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6496.768127] Lustre: Skipped 1 previous similar message [ 6501.870600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6501.878518] Lustre: Skipped 2 previous similar messages [ 6502.304668] Lustre: server umount lustre-MDT0000 complete [ 6506.650952] LustreError: 150323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847047 with bad export cookie 1701944532707018046 [ 6506.653447] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6506.657663] LustreError: 150323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6506.977955] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6506.989842] Lustre: Skipped 1 previous similar message [ 6507.159815] Lustre: server umount lustre-MDT0001 complete [ 6513.618631] Lustre: server umount lustre-OST0000 complete [ 6542.369665] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 6555.263984] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 6593.651970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.935575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6593.960976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6594.035590] Lustre: lustre-MDT0000: new disk, initializing [ 6594.107374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.131213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6599.397483] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6611.750996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.879174] Lustre: 157119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6611.946690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6611.951355] Lustre: Skipped 1 previous similar message [ 6612.039732] Lustre: lustre-MDT0001: new disk, initializing [ 6612.200079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6612.244510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6612.256682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6617.518488] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6626.495740] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6631.460748] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6638.137080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6638.471713] Lustre: lustre-OST0000: new disk, initializing [ 6638.474453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6638.480454] Lustre: 159086:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6638.610518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6645.251585] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6646.392042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6646.412614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6646.604083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6655.257792] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6660.504566] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6660.787405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6665.026947] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6665.197851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6667.233404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6667.240284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.245184] LustreError: Skipped 1 previous similar message [ 6667.258169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6672.355381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6672.371601] Lustre: Skipped 1 previous similar message [ 6673.281523] Lustre: server umount lustre-OST0000 complete [ 6677.481861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6677.490847] Lustre: Skipped 2 previous similar messages [ 6682.592865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6682.597965] Lustre: Skipped 3 previous similar messages [ 6683.518430] Lustre: server umount lustre-MDT0000 complete [ 6687.101787] LustreError: 157111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847228 with bad export cookie 1701944532707019005 [ 6687.105538] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6687.114158] LustreError: 157111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6687.496981] Lustre: server umount lustre-MDT0001 complete [ 6691.254393] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 12:13:50 (1787847230) [ 6693.249893] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6695.131037] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 12:13:54 (1787847234) [ 6704.096029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6704.703472] LustreError: 161499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.727522] LustreError: 161499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6704.780854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6709.123629] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6710.241106] LustreError: 161500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6714.336502] LustreError: 161499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.726308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6724.305940] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6734.913284] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6741.576068] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6749.060387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6749.338869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6749.341170] Lustre: Skipped 1 previous similar message [ 6755.301576] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6764.310260] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6784.482500] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6784.486570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6784.499076] Lustre: Skipped 1 previous similar message [ 6784.507157] Lustre: Skipped 2 previous similar messages [ 6787.406405] Lustre: server umount lustre-MDT0000 complete [ 6789.600504] LustreError: 161500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6789.618160] LustreError: 161500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6791.195580] LustreError: 161480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847332 with bad export cookie 1701944532707019887 [ 6791.202216] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6791.205544] LustreError: 161480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6794.732486] LustreError: 161494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6794.754992] LustreError: 161494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6797.926405] Lustre: server umount lustre-MDT0001 complete [ 6804.345158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6816.429333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.593257] LustreError: 165362:0:(llog.c:1655:llog_backup()) MGC192.168.201.111@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6816.599764] Lustre: 165362:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.111@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6821.344856] LustreError: 165367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6821.355846] LustreError: 165367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6841.735659] LustreError: 165369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6841.750831] LustreError: 165369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6857.195610] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6857.217603] Lustre: Skipped 4 previous similar messages [ 6876.570762] LustreError: 165368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6876.601163] LustreError: 165368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 6877.663357] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6877.668261] Lustre: 165362:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.111@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6918.657301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6922.578663] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6923.752293] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6930.903993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.032518] LustreError: 148552:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004c908da3 x1874693362678656/t0(0) o250->MGC192.168.201.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6941.323481] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 6941.675695] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6948.998431] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6950.644317] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6956.178281] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6957.814298] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6977.506307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6977.507527] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.526825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6977.528527] Lustre: Skipped 3 previous similar messages [ 6977.543636] Lustre: Skipped 2 previous similar messages [ 6980.608136] Lustre: server umount lustre-MDT0000 complete [ 6982.625229] LustreError: 165368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6982.648068] LustreError: 165368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 6984.785323] LustreError: 161481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847525 with bad export cookie 1701944532707026341 [ 6984.792852] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6984.799624] LustreError: 161481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6985.218632] Lustre: server umount lustre-MDT0001 complete [ 6999.264598] Lustre: server umount lustre-OST0000 complete [ 7018.386867] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 7027.780221] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 7060.107874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.314910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7060.333677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7060.398362] Lustre: lustre-MDT0000: new disk, initializing [ 7060.479651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7060.490778] Lustre: Skipped 1 previous similar message [ 7060.509871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7065.216960] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7075.408414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7075.555124] Lustre: 170469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7075.856686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7080.256476] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7088.930439] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7094.592769] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7101.691740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7102.002795] Lustre: lustre-OST0000: new disk, initializing [ 7102.008672] Lustre: Skipped 1 previous similar message [ 7102.018499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7102.022752] Lustre: Skipped 2 previous similar messages [ 7102.027100] Lustre: 172437:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7107.918795] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7110.738075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7110.747287] Lustre: Skipped 1 previous similar message [ 7110.750950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7110.867346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7116.926259] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7121.340197] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7121.523629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7125.540391] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7125.743847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7131.618275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7131.622616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7131.633168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7131.636175] LustreError: Skipped 1 previous similar message [ 7131.650536] Lustre: Skipped 2 previous similar messages [ 7133.627696] Lustre: server umount lustre-OST0000 complete [ 7142.822450] Lustre: server umount lustre-MDT0000 complete [ 7146.051327] LustreError: 170461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847687 with bad export cookie 1701944532707043176 [ 7146.071466] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7146.492863] Lustre: server umount lustre-MDT0001 complete [ 7156.060596] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 12:21:34 (1787847694) [ 7161.528826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.005842] LustreError: 174847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7165.365757] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7173.398267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7177.975373] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7185.038839] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7191.216343] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7198.185050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7204.737243] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7211.345427] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7220.181990] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 12:22:39 (1787847759) [ 7225.312854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7225.325828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7225.345557] Lustre: Skipped 2 previous similar messages [ 7230.433946] Lustre: server umount lustre-OST0000 complete [ 7235.052572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7235.065976] LustreError: Skipped 1 previous similar message [ 7249.376157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7249.601540] Lustre: server umount lustre-MDT0000 complete [ 7252.914763] LustreError: 174827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847794 with bad export cookie 1701944532707044051 [ 7252.932896] LustreError: 174827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7252.932990] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7253.681844] Lustre: server umount lustre-MDT0001 complete [ 7265.152423] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 7268.099170] Key type lgssc unregistered [ 7268.428110] LNet: 178579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7268.436575] LNetError: 178579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7268.461820] LNet: Removed LNI 192.168.201.111@tcp [ 7269.329545] Key type .llcrypt unregistered [ 7269.335268] Key type ._llcrypt unregistered [ 7290.416372] Key type ._llcrypt registered [ 7290.418961] Key type .llcrypt registered [ 7290.514660] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 7291.946114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7292.002540] alg: No test for adler32 (adler32-zlib) [ 7293.042608] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 7293.261963] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 7294.919199] Key type lgssc registered [ 7296.347881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7304.950792] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7304.979650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7320.672586] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7325.793653] LustreError: 179825:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.111@tcp: failed processing log, type 4: rc = -110 [ 7330.918823] LustreError: 179825:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7330.943197] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7330.949470] LustreError: MGC192.168.201.111@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7330.966968] LustreError: 179825:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7330.974565] LustreError: 179825:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7330.986362] LustreError: 179825:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7331.094333] Lustre: server umount lustre-OST0000 complete [ 7331.100223] LustreError: 179825:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7339.820085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7341.316577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7345.817183] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7351.783269] LustreError: 180324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7351.813384] LustreError: 180324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7356.755920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.900774] LustreError: 180325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7357.574793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7362.898671] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7372.011514] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7379.072169] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7387.620310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7387.819992] LustreError: 182142:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7387.837918] Lustre: 182142:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.111@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7388.044135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7397.090974] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7406.433916] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7412.712055] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.739819] Lustre: Skipped 1 previous similar message [ 7412.743722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7412.760650] Lustre: Skipped 2 previous similar messages [ 7415.441103] Lustre: server umount lustre-MDT0000 complete [ 7417.839688] LustreError: 180324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7417.871038] LustreError: 180324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7420.951118] LustreError: 180305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847962 with bad export cookie 14650628622518986629 [ 7420.964797] LustreError: 180305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7420.973292] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7421.613677] Lustre: server umount lustre-MDT0001 complete [ 7436.243646] Lustre: server umount lustre-OST0000 complete [ 7448.818334] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 7461.056872] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 7501.076295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7501.311168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7501.335979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7501.415386] Lustre: lustre-MDT0000: new disk, initializing [ 7501.482906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7501.526747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7507.274984] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7519.344327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.622550] Lustre: 185669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7519.750681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7519.761815] Lustre: Skipped 1 previous similar message [ 7519.966874] Lustre: lustre-MDT0001: new disk, initializing [ 7520.203690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7520.261529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7520.284632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7527.633827] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7539.161946] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7547.755678] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7555.123382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7555.365335] Lustre: lustre-OST0000: new disk, initializing [ 7555.368873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7555.374821] Lustre: 187635:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7555.443566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.240962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7561.251657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7561.347338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7562.015608] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7573.635973] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7579.947238] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7580.337578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7586.375162] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7586.735794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7591.928981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.930905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7591.945095] Lustre: Skipped 1 previous similar message [ 7591.946473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7591.968912] Lustre: Skipped 1 previous similar message [ 7597.027340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7597.038865] Lustre: Skipped 1 previous similar message [ 7602.148132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7602.156527] Lustre: Skipped 1 previous similar message [ 7604.191215] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7604.608836] Lustre: server umount lustre-OST0000 complete [ 7612.390555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7612.401961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7612.419052] Lustre: Skipped 1 previous similar message [ 7612.420375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7615.648927] Lustre: server umount lustre-MDT0000 complete [ 7617.504288] LustreError: 188978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7617.513775] LustreError: 188978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7620.083056] LustreError: 185660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848161 with bad export cookie 14650628622518987336 [ 7620.095165] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7620.101315] LustreError: 185660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7620.508919] Lustre: server umount lustre-MDT0001 complete [ 7630.237239] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:29:29 (1787848169) [ 7652.465612] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 7656.132036] Key type lgssc unregistered [ 7656.571244] LNet: 190588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7656.574615] LNetError: 190588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7657.638272] LNet: Removed LNI 192.168.201.111@tcp [ 7658.751168] Key type .llcrypt unregistered [ 7658.757084] Key type ._llcrypt unregistered [ 7679.694933] Key type ._llcrypt registered [ 7679.696893] Key type .llcrypt registered [ 7679.856640] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 7680.882329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7680.982487] alg: No test for adler32 (adler32-zlib) [ 7682.133544] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 7682.430384] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 7684.151256] Key type lgssc registered [ 7685.766478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7695.089744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7695.109878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7696.732876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7702.558841] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7706.097641] LustreError: 191807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7706.120697] LustreError: 191807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7711.207721] LustreError: 191808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7712.262824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7712.627766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7718.107445] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7727.728848] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7734.496470] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7743.166391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7743.382745] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7743.391251] LustreError: 193672:0:(llog.c:1696:llog_backup()) MGC192.168.201.111@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7743.401972] Lustre: 193672:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.111@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7743.558502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7752.607494] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7764.697923] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7769.058170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7769.067950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7769.179158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7771.616822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7771.619546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7771.664290] Lustre: Skipped 1 previous similar message [ 7771.682164] Lustre: Skipped 1 previous similar message [ 7776.745879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7776.763122] Lustre: Skipped 3 previous similar messages [ 7781.859500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.885483] Lustre: Skipped 2 previous similar messages [ 7782.371115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7782.801344] Lustre: server umount lustre-MDT0000 complete [ 7786.977872] LustreError: 191808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7787.003035] LustreError: 191808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7789.122402] LustreError: 191789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848330 with bad export cookie 9312363078210499647 [ 7789.141492] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7789.145168] LustreError: 191789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7789.459296] Lustre: server umount lustre-MDT0001 complete [ 7805.995156] Lustre: server umount lustre-OST0000 complete [ 7818.228651] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 7832.194819] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 7877.632967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7877.856439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7877.910681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7878.022616] Lustre: lustre-MDT0000: new disk, initializing [ 7878.134940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7878.150835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7883.663656] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7895.150963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7895.319403] Lustre: 197205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7895.367339] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7895.370416] Lustre: Skipped 1 previous similar message [ 7895.374779] LustreError: 197987:0:(llog.c:1696:llog_backup()) MGC192.168.201.111@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7895.382392] LustreError: 197987:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7895.387760] Lustre: 197987:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.111@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7895.408894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7895.412463] Lustre: Skipped 1 previous similar message [ 7895.475893] Lustre: lustre-MDT0001: new disk, initializing [ 7895.586808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7895.639938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7895.653195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7900.695554] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7911.398129] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7917.708584] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7925.271640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7925.756450] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7925.759234] Lustre: Skipped 1 previous similar message [ 7925.764854] LustreError: 199150:0:(llog.c:1696:llog_backup()) MGC192.168.201.111@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7925.771747] LustreError: 199150:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7925.777757] Lustre: 199150:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.111@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7925.788247] Lustre: 199150:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7925.818353] Lustre: lustre-OST0000: new disk, initializing [ 7925.828920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7925.838851] Lustre: 199173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7925.978994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7931.483305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7931.488869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7931.627579] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7933.609543] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 7943.235622] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7948.477946] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7948.758398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7953.522797] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7953.718423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7957.477676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7957.480523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7957.491119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7957.526330] Lustre: Skipped 1 previous similar message [ 7957.560215] LustreError: Skipped 1 previous similar message [ 7961.765917] Lustre: server umount lustre-OST0000 complete [ 7967.205032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7967.213375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7967.235137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7967.240434] Lustre: Skipped 1 previous similar message [ 7972.036367] Lustre: server umount lustre-MDT0000 complete [ 7972.834503] LustreError: 197217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7972.863709] LustreError: 197217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7975.846508] LustreError: 197196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848517 with bad export cookie 9312363078210500368 [ 7975.849291] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7975.862455] LustreError: 197196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7976.327436] Lustre: server umount lustre-MDT0001 complete [ 7985.038512] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:35:24 (1787848524) [ 8010.100303] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8025.291398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8026.020437] LustreError: 202325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8026.122132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8034.205990] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8036.320192] LustreError: 202326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8036.339934] LustreError: 202326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8044.311936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.036819] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8053.653598] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8061.087029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8061.402558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8061.414030] Lustre: Skipped 1 previous similar message [ 8068.520976] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8078.118098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8078.379183] Lustre: lustre-OST0001: new disk, initializing [ 8078.385650] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8078.394197] Lustre: 204633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8085.275379] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8088.156754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8088.168316] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8088.257946] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8096.151667] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8106.351409] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8108.719630] Lustre: DEBUG MARKER: ADD OST3 [ 8114.002832] loop: module loaded [ 8114.089193] loop0: detected capacity change from 0 to 204800000 [ 8114.523680] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8114.555353] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8114.575285] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8122.195579] Lustre: DEBUG MARKER: STOP OST3 [ 8122.461726] loop0: detected capacity change from 0 to 204800000 [ 8128.217105] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8128.501595] Lustre: lustre-OST0003: new disk, initializing [ 8128.505700] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8128.517271] Lustre: 206695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8128.730286] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8128.736573] Lustre: Skipped 1 previous similar message [ 8130.564919] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8130.775502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8130.802813] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8130.865101] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8131.284173] Lustre: Failing over lustre-OST0003 [ 8131.440785] Lustre: server umount lustre-OST0003 complete [ 8132.577340] LustreError: 206874:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8135.609202] LustreError: 203813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8135.627408] LustreError: 203813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8135.910508] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8136.160505] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8136.169503] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8136.178757] Lustre: Skipped 1 previous similar message [ 8147.155482] Lustre: DEBUG MARKER: START OST3 again [ 8154.336991] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8154.656602] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8155.958884] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8156.099644] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8156.099687] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8162.442450] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8185.269967] Lustre: Failing over lustre-OST0003 [ 8185.333080] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8185.336691] Lustre: lustre-OST0003: Not available for connect from 0@lo (stopping) [ 8185.349861] Lustre: Skipped 2 previous similar messages [ 8185.375082] Lustre: Skipped 2 previous similar messages [ 8185.618725] Lustre: server umount lustre-OST0003 complete [ 8190.455462] LustreError: 203812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8190.496075] LustreError: 203812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8205.802638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8205.824816] Lustre: Skipped 1 previous similar message [ 8210.601044] Lustre: server umount lustre-MDT0000 complete [ 8214.517181] LustreError: 202307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848755 with bad export cookie 9312363078210501250 [ 8214.530900] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8214.534370] LustreError: 202307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8219.051772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8219.059948] Lustre: Skipped 7 previous similar messages [ 8220.966624] Lustre: server umount lustre-MDT0001 complete [ 8224.599200] Lustre: server umount lustre-OST0000 complete [ 8228.623601] Lustre: server umount lustre-OST0001 complete [ 8235.615529] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 8243.691395] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8283.493925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8283.710954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8283.723908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8283.766891] Lustre: lustre-MDT0000: new disk, initializing [ 8283.820253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8283.824589] Lustre: Skipped 1 previous similar message [ 8283.837852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8288.824908] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8302.247071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8302.441635] Lustre: 212248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8302.488505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8302.492687] Lustre: Skipped 1 previous similar message [ 8302.555179] Lustre: lustre-MDT0001: new disk, initializing [ 8302.754669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8302.768793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8308.962092] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8320.609675] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8326.584154] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8333.954044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8334.169583] Lustre: 214218:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8336.168765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8336.185869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8336.383337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8341.772456] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8353.013674] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8359.501911] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8359.772658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8365.801917] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8366.015574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8372.192305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8372.194396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8372.221377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8372.248979] Lustre: Skipped 5 previous similar messages [ 8374.288931] Lustre: server umount lustre-OST0000 complete [ 8384.500876] Lustre: server umount lustre-MDT0000 complete [ 8387.571950] LustreError: 212261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8387.603342] LustreError: 212261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8389.414582] LustreError: 214217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848930 with bad export cookie 9312363078210503546 [ 8389.419024] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8389.431990] LustreError: 214217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8390.044944] Lustre: server umount lustre-MDT0001 complete [ 8394.218511] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 12:42:13 (1787848933) [ 8414.485552] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 8425.245122] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8465.832341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.192221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8466.227990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8466.238419] Lustre: Skipped 1 previous similar message [ 8466.403313] Lustre: lustre-MDT0000: new disk, initializing [ 8466.411931] Lustre: Skipped 1 previous similar message [ 8466.544243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8472.457384] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8483.849579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.042704] Lustre: 218830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8484.423874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8490.337929] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8501.439903] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8508.734331] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8516.221776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8516.581958] Lustre: 220797:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8517.739301] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8517.899897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8523.595553] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8534.082115] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8539.889794] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8540.158268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8546.607103] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8547.027706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8553.959577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8553.960241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8554.002860] LustreError: Skipped 2 previous similar messages [ 8554.012102] Lustre: Skipped 2 previous similar messages [ 8554.015249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8554.015257] Lustre: Skipped 4 previous similar messages [ 8555.706957] Lustre: server umount lustre-OST0000 complete [ 8567.573637] Lustre: server umount lustre-MDT0000 complete [ 8569.315452] LustreError: 221541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8569.342048] LustreError: 221541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8571.067853] LustreError: 218821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849112 with bad export cookie 9312363078210504428 [ 8571.073103] LustreError: 218821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8571.075270] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8571.699706] Lustre: server umount lustre-MDT0001 complete [ 8581.211635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8581.626313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8581.632983] Lustre: Skipped 5 previous similar messages [ 8585.895429] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8593.777750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8598.141887] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8605.019302] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8610.660121] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8617.347310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8623.758342] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8630.125883] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8640.480400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8645.775376] Lustre: server umount lustre-OST0000 complete [ 8655.484624] Lustre: server umount lustre-MDT0000 complete [ 8659.491852] LustreError: 222998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849200 with bad export cookie 9312363078210505310 [ 8659.498524] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8659.512607] LustreError: 222998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8659.991585] Lustre: server umount lustre-MDT0001 complete [ 8771.570959] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 8774.364498] Key type lgssc unregistered [ 8774.626201] LNet: 226684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8774.636596] LNetError: 226684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8774.658925] LNet: Removed LNI 192.168.201.111@tcp [ 8775.958124] Key type .llcrypt unregistered [ 8775.962098] Key type ._llcrypt unregistered [ 8806.874761] Key type ._llcrypt registered [ 8806.878305] Key type .llcrypt registered [ 8807.097541] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 8825.816971] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 8826.985179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8827.017761] alg: No test for adler32 (adler32-zlib) [ 8828.272513] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 8828.569400] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 8830.266167] Key type lgssc registered [ 8831.691976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8868.963459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8868.991795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.299837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8870.356577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8870.431100] Lustre: lustre-MDT0000: new disk, initializing [ 8870.513715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8870.540549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8876.018552] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8888.786752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8888.977789] Lustre: 230037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8889.048767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8889.052272] Lustre: Skipped 1 previous similar message [ 8889.107883] Lustre: lustre-MDT0001: new disk, initializing [ 8889.180092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8889.212186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8889.228185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8894.520199] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8905.521297] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8912.250333] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8920.093608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8920.626735] Lustre: lustre-OST0000: new disk, initializing [ 8920.631391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8920.639181] Lustre: 232007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8920.735581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8927.316457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8927.330685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8927.473445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8927.656682] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 8939.239493] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8944.294303] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8944.471304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8949.255285] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8949.456720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8953.317875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8953.320187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8953.322257] LustreError: Skipped 1 previous similar message [ 8953.335156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8953.349592] Lustre: Skipped 1 previous similar message [ 8953.364863] Lustre: Skipped 1 previous similar message [ 8958.125564] Lustre: server umount lustre-OST0000 complete [ 8966.114628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8966.133065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8966.177655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8968.674092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8968.678402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8970.871548] Lustre: server umount lustre-MDT0000 complete [ 8978.404738] LustreError: 231120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.447590] LustreError: 231120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8980.468138] LustreError: 230834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849521 with bad export cookie 6243569008565028979 [ 8980.475429] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8980.478313] LustreError: 230834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8980.864811] Lustre: server umount lustre-MDT0001 complete [ 9013.296538] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9025.752585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9026.509972] LustreError: 234964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9026.599253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9030.945829] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9031.648318] LustreError: 234963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9036.791978] LustreError: 234964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9039.546877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9039.907874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9044.433187] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9047.503996] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9053.739877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9054.190508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9060.134257] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9069.105604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9069.354681] Lustre: lustre-OST0001: new disk, initializing [ 9069.358282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9069.366580] Lustre: 237269:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9070.303618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9070.315707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9070.550464] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9075.762919] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9087.382541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9097.064413] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9110.727280] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:54:09 (1787849649) [ 9116.648260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9116.660252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9116.677180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9116.687665] Lustre: Skipped 1 previous similar message [ 9122.788134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9122.810463] Lustre: Skipped 7 previous similar messages [ 9127.391969] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9127.837741] Lustre: server umount lustre-MDT0000 complete [ 9127.912524] LustreError: 235691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9127.940397] LustreError: 235691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9132.391872] LustreError: 234943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849673 with bad export cookie 6243569008565029861 [ 9132.423490] LustreError: 234943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9132.426554] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9132.826845] Lustre: server umount lustre-MDT0001 complete [ 9147.919506] Lustre: server umount lustre-OST0000 complete [ 9149.599122] Lustre: 228260:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849674/real 1787849674] req@ffff8d0241087840 x1874695940067968/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9149.623161] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9149.645037] Lustre: Skipped 3 previous similar messages [ 9152.615958] Lustre: server umount lustre-OST0001 complete [ 9161.898118] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 9173.000435] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9211.698812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9211.971557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9212.025121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9212.126204] Lustre: lustre-MDT0000: new disk, initializing [ 9212.230723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9212.237304] Lustre: Skipped 1 previous similar message [ 9212.259302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9216.680539] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9229.471240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9229.601635] Lustre: 241219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9229.658992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9229.667541] Lustre: Skipped 1 previous similar message [ 9229.757629] Lustre: lustre-MDT0001: new disk, initializing [ 9229.877933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9229.892773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9234.410228] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9243.569287] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9248.997994] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9255.903859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9256.202890] Lustre: lustre-OST0000: new disk, initializing [ 9256.207767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9256.212523] Lustre: 243187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9256.294372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9256.299704] Lustre: Skipped 1 previous similar message [ 9258.341518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9258.358156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9258.563275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9262.731227] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9273.055221] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9279.316343] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9279.528579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9283.814448] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9284.012163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9289.197282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9289.199521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9289.223263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9289.237089] LustreError: Skipped 1 previous similar message [ 9292.519481] Lustre: server umount lustre-OST0000 complete [ 9303.439812] Lustre: server umount lustre-MDT0000 complete [ 9304.552782] LustreError: 241225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9304.569178] LustreError: 241225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9307.100131] LustreError: 241212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849848 with bad export cookie 6243569008565032003 [ 9307.100976] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9307.115556] LustreError: 241212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9307.577122] Lustre: server umount lustre-MDT0001 complete [ 9317.836692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9324.017467] LustreError: 245404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9324.044337] LustreError: 245404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9326.243880] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9343.093475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9343.459143] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9343.485043] Lustre: Skipped 3 previous similar messages [ 9343.900889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9343.906383] Lustre: Skipped 1 previous similar message [ 9351.371488] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9361.016816] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9368.344617] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9375.491238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9383.037305] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9390.404567] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9536.998091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9537.005916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9537.017819] Lustre: Skipped 3 previous similar messages [ 9537.021684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9551.327327] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9551.580836] Lustre: server umount lustre-OST0000 complete [ 9559.009727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9559.022104] LustreError: Skipped 1 previous similar message [ 9561.467487] Lustre: server umount lustre-MDT0000 complete [ 9564.645079] LustreError: 245403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9564.668414] LustreError: 245403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9565.688946] LustreError: 245385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850106 with bad export cookie 6243569008565032885 [ 9565.695847] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9565.707632] LustreError: 245385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9565.933990] Lustre: server umount lustre-MDT0001 complete [ 9660.830414] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [ 9664.444765] Key type lgssc unregistered [ 9664.753270] LNet: 249041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9664.769470] LNetError: 249041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9664.807234] LNet: Removed LNI 192.168.201.111@tcp [ 9665.889500] Key type .llcrypt unregistered [ 9665.893634] Key type ._llcrypt unregistered [ 9694.280535] Key type ._llcrypt registered [ 9694.286937] Key type .llcrypt registered [ 9694.421547] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [ 9711.201632] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9712.574268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9712.703828] alg: No test for adler32 (adler32-zlib) [ 9714.403149] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [ 9714.700730] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [ 9716.535909] Key type lgssc registered [ 9718.253783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9767.402545] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9778.579955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9778.617399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9780.011858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9780.062082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9780.164455] Lustre: lustre-MDT0000: new disk, initializing [ 9780.266027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9780.285853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9785.817573] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9797.372739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9797.513709] Lustre: 253116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9797.612246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9797.617297] Lustre: Skipped 1 previous similar message [ 9797.711172] Lustre: lustre-MDT0001: new disk, initializing [ 9797.786251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9797.834604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9797.850354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9802.918801] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9808.556651] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9816.814268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9817.051460] Lustre: lustre-OST0000: new disk, initializing [ 9817.055457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9817.063727] Lustre: 254747:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9817.196146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9822.736497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9822.749981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9822.834125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9823.802415] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9837.041878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9837.276522] Lustre: lustre-OST0001: new disk, initializing [ 9837.279966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9837.285252] Lustre: 255620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9837.417108] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9844.688805] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9847.448448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9847.463829] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9847.529553] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9855.945699] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9865.158262] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9877.474861] Lustre: server umount lustre-MDT0000 complete [ 9878.509388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9878.526020] Lustre: Skipped 2 previous similar messages [ 9881.124656] LustreError: 253107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850422 with bad export cookie 14332443450970958002 [ 9881.128940] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9881.149527] LustreError: 253107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9881.615730] Lustre: server umount lustre-MDT0001 complete [ 9896.110505] Lustre: server umount lustre-OST0000 complete [ 9898.911345] Lustre: 250623:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850424/real 1787850424] req@ffff8d0242996d00 x1874696869064576/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9898.939739] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9898.947900] Lustre: Skipped 1 previous similar message [ 9899.692852] Lustre: server umount lustre-OST0001 complete [ 9912.996228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9923.191959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9931.055313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9940.254991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9959.516450] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [ 9969.465890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9969.509165] Lustre: lustre-MDT0000: reset Object Index mappings [ 9969.953184] LustreError: 260629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9969.972155] LustreError: 260629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9970.077173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9974.825059] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9975.265654] LustreError: 260630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9975.301281] LustreError: 260630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9980.385229] LustreError: 260629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9983.954458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9983.971784] Lustre: lustre-MDT0001: reset Object Index mappings [ 9984.654448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9989.806453] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [ 9993.224696] Lustre: 261772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9999.674352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10007.031069] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10009.134301] LustreError: 262125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10014.205353] LustreError: 262125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10015.257245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10017.490462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10017.972882] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10017.985617] Lustre: Skipped 1 previous similar message [10023.440933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10026.506888] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10035.452662] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10039.494118] Lustre: 263645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10049.000114] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10049.008989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10049.030391] Lustre: Skipped 3 previous similar messages [10049.043703] Lustre: Skipped 3 previous similar messages [10052.653112] Lustre: server umount lustre-MDT0000 complete [10054.123702] LustreError: 260629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10054.142982] LustreError: 260629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10056.788022] LustreError: 261374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850597 with bad export cookie 14332443450970964792 [10056.795185] LustreError: 261374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10056.796767] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10057.151697] Lustre: server umount lustre-MDT0001 complete [10070.042614] Lustre: server umount lustre-OST0000 complete [10085.671533] Lustre: server umount lustre-OST0001 complete [10108.875264] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:10:47 (1787850647) [10129.228282] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [10138.964808] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10175.783159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10176.166895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10176.220531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10176.399756] Lustre: lustre-MDT0000: new disk, initializing [10176.514497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10176.553619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10181.546403] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10192.678547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10192.853771] Lustre: 267902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10192.939278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10192.944600] Lustre: Skipped 1 previous similar message [10193.062293] Lustre: lustre-MDT0001: new disk, initializing [10193.286935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10193.307339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10200.570165] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10210.997921] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10217.282836] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10224.216420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10224.679217] Lustre: lustre-OST0000: new disk, initializing [10224.685297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10224.700833] Lustre: 269872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10226.562932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10226.579692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10226.828148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10233.521144] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10248.138266] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10253.528249] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10253.764697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10260.098429] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10260.375410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10262.507566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10262.508362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10262.517532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10262.537095] LustreError: Skipped 1 previous similar message [10262.563074] Lustre: Skipped 1 previous similar message [10267.616820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10267.635251] Lustre: Skipped 1 previous similar message [10268.836471] Lustre: server umount lustre-OST0000 complete [10275.298941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10275.314452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10275.338338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10275.347326] Lustre: Skipped 1 previous similar message [10279.623154] Lustre: server umount lustre-MDT0000 complete [10282.980063] LustreError: 267914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10283.001103] LustreError: 267914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10284.212390] LustreError: 267895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850825 with bad export cookie 14332443450970966885 [10284.215085] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10284.225242] LustreError: 267895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10284.734126] Lustre: server umount lustre-MDT0001 complete [10315.418756] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10329.962411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10330.728832] LustreError: 272830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10330.834036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10330.841562] Lustre: Skipped 2 previous similar messages [10336.426500] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10348.938232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.070597] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10360.526625] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10367.693612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10375.085583] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10384.677201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10384.851672] Lustre: lustre-OST0001: new disk, initializing [10384.855769] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10384.872489] Lustre: 275139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10386.494887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10386.504540] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10386.604163] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10391.331477] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10400.567978] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10403.856945] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10408.744843] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10408.840817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10415.420551] Lustre: server umount lustre-MDT0000 complete [10415.590436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10415.608489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10415.644567] Lustre: Skipped 1 previous similar message [10415.671809] LustreError: 272830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10415.696798] LustreError: 272830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10419.108974] LustreError: 272811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850960 with bad export cookie 14332443450970967767 [10419.113663] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10419.118887] LustreError: 272811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10419.349645] Lustre: server umount lustre-MDT0001 complete [10433.041617] Lustre: server umount lustre-OST0000 complete [10446.207792] Lustre: server umount lustre-OST0001 complete [10465.607203] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10478.288261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10478.729392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10478.734415] Lustre: Skipped 3 previous similar messages [10483.297619] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10494.026473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10500.288976] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10503.975942] Lustre: 279331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10511.995033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10520.482680] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10523.817275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10530.292572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10535.935079] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10537.525393] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10545.269918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10549.407467] Lustre: 281208:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10554.771164] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10554.984921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10564.013404] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:18:23 (1787851103) [10581.474123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10581.481685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10581.491841] Lustre: Skipped 3 previous similar messages [10581.502272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10581.504320] Lustre: Skipped 2 previous similar messages [10586.710545] Lustre: server umount lustre-MDT0000 complete [10587.117947] LustreError: 278189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10587.131674] LustreError: 278189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [10591.183552] LustreError: 281211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851132 with bad export cookie 14332443450970974067 [10591.188116] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10591.198596] LustreError: 281211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10592.226721] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10592.255884] Lustre: Skipped 4 previous similar messages [10598.370906] Lustre: server umount lustre-MDT0001 complete [10617.311956] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10617.404802] Lustre: server umount lustre-OST0000 complete [10622.172114] Lustre: server umount lustre-OST0001 complete [10629.403365] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [10640.907951] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10695.189601] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [10706.597621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10706.953222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10707.045884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10707.163950] Lustre: lustre-MDT0000: new disk, initializing [10707.294277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10711.838862] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10723.559853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10723.810805] Lustre: 285473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10724.179766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10729.741291] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10743.768601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10744.110466] Lustre: 287108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10744.209192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10744.214848] Lustre: Skipped 5 previous similar messages [10746.107253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10751.594478] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10763.658298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10763.905775] Lustre: 287978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10765.678439] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10765.700360] Lustre: Skipped 1 previous similar message [10765.917551] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10773.858228] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10788.444306] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10795.114703] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10795.121293] Lustre: Skipped 1 previous similar message [10813.344656] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 13:22:32 (1787851352) [10816.480704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10816.496425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10816.512017] Lustre: Skipped 5 previous similar messages [10816.517042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10816.521381] Lustre: Skipped 2 previous similar messages [10830.305922] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10830.484737] Lustre: server umount lustre-MDT0000 complete [10834.174817] LustreError: 285467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851375 with bad export cookie 14332443450970981956 [10834.176113] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10834.188305] LustreError: 285467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10834.523900] Lustre: server umount lustre-MDT0001 complete [10849.181808] Lustre: server umount lustre-OST0000 complete [10852.831221] Lustre: 250620:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851378/real 1787851378] req@ffff8d0346d4b840 x1874696869385472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787851394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10853.944281] Lustre: server umount lustre-OST0001 complete [10862.496738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10862.949669] LustreError: 290231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10862.976422] LustreError: 290231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10867.949838] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10877.285731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10882.747534] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10890.467218] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10896.101916] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10902.905140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10910.678786] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10919.139567] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10939.880793] loop1: detected capacity change from 0 to 204800000 [10940.101964] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10940.139441] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10940.165724] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10945.084932] loop1: detected capacity change from 0 to 204800000 [10950.560240] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10950.679405] LustreError: 290221:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs2320: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10950.690990] LustreError: 290221:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs2320 [10950.695710] LustreError: 290221:0:(mgs_handler.c:517:mgs_target_reg()) fs2320-MDT0001: Error checking target index: rc = -13 [10950.706631] LustreError: fs2320-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10950.720956] LustreError: 293948:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10950.727718] LustreError: 293948:0:(tgt_mount.c:2129:server_put_super()) no obd fs2320-MDT0001 [10950.737324] LustreError: 293948:0:(tgt_mount.c:116:server_deregister_mount()) fs2320-MDT0001 not registered [10950.807657] Lustre: server umount fs2320-MDT0001 complete [10950.812365] LustreError: 293948:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10961.442678] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10961.785185] Lustre: Setting parameter fs2320-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs2320-MDT0001 [10961.920822] Lustre: srv-fs2320-MDT0001: No data found on store. Initialize space. [10961.936271] Lustre: Skipped 4 previous similar messages [10962.122753] Lustre: fs2320-MDT0001: new disk, initializing [10962.137210] Lustre: Skipped 3 previous similar messages [10962.296997] Lustre: srv-fs2320-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10962.305425] Lustre: 294417:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2320-MDT0001-mdtlov: can't create new llog: rc = -115 [10962.319733] LustreError: 294417:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2320-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10967.088778] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [10975.180081] Lustre: server umount fs2320-MDT0001 complete [10988.548811] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10992.856476] loop1: detected capacity change from 0 to 204800000 [11000.633384] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11001.297607] Lustre: 295986:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2320-MDT0001-mdtlov: can't create new llog: rc = -115 [11001.311750] LustreError: 295986:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2320-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11006.450141] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11012.348434] Lustre: server umount fs2320-MDT0001 complete [11024.362416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11024.365181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11024.365192] Lustre: Skipped 11 previous similar messages [11024.400033] Lustre: Skipped 6 previous similar messages [11030.002986] Lustre: server umount lustre-MDT0000 complete [11039.413868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11039.683695] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11045.370110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11046.144420] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11055.210953] loop1: detected capacity change from 0 to 204800000 [11061.708898] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11062.392750] Lustre: 298432:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs2320-MDT0001-mdtlov: can't create new llog: rc = -115 [11062.411136] LustreError: 298432:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs2320-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11067.517372] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11073.287946] Lustre: server umount fs2320-MDT0001 complete [11101.153747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11101.171759] LustreError: Skipped 1 previous similar message [11101.175711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11101.177871] Lustre: Skipped 6 previous similar messages [11106.583666] Lustre: server umount lustre-OST0000 complete [11117.580445] Lustre: server umount lustre-MDT0000 complete [11121.168060] LustreError: 290211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851662 with bad export cookie 14332443450970984637 [11121.170043] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11121.175612] LustreError: 290211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11121.554885] Lustre: server umount lustre-MDT0001 complete [11132.896845] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11136.443074] Key type lgssc unregistered [11136.788883] LNet: 300779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11136.799774] LNetError: 300779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11136.815304] LNet: Removed LNI 192.168.201.111@tcp [11137.774228] Key type .llcrypt unregistered [11137.778439] Key type ._llcrypt unregistered [11164.627750] Key type ._llcrypt registered [11164.629521] Key type .llcrypt registered [11164.791634] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11166.670543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11166.960904] alg: No test for adler32 (adler32-zlib) [11168.376738] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [11168.873864] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11170.735228] Key type lgssc registered [11172.899885] Lustre: Echo OBD driver; http://www.lustre.org/ [11182.158280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11182.190047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11183.901967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11189.408662] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11193.325056] LustreError: 302055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11193.352242] LustreError: 302055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11194.357134] LustreError: 302054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11199.462690] LustreError: 302055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11199.909643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11200.707705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11206.325498] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11218.016143] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11225.873075] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11234.032612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11234.490429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11242.606470] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11245.731703] LustreError: 303885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11250.482700] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11251.177410] LustreError: 303888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11259.362448] LustreError: 303885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11259.404954] LustreError: 303885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11278.236997] LustreError: 303888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.280259] LustreError: 303888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11282.801446] loop1: detected capacity change from 0 to 204800000 [11283.177564] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11283.195701] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11283.209882] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11290.277667] loop1: detected capacity change from 0 to 204800000 [11297.898935] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11298.179293] Lustre: fs4241-OST0000: new disk, initializing [11298.187152] Lustre: srv-fs4241-OST0000: No data found on store. Initialize space. [11298.201092] Lustre: 305812:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11298.382355] Lustre: fs4241-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11305.560380] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11310.579353] LustreError: 304371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11310.616554] LustreError: 304371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11314.686257] Lustre: server umount fs4241-OST0000 complete [11320.863466] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11329.482765] loop1: detected capacity change from 0 to 204800000 [11331.052366] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11335.644554] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11335.933484] Lustre: fs4241-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11342.366251] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11352.201080] loop2: detected capacity change from 0 to 204800000 [11352.643597] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11352.660111] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11352.685356] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11358.963400] loop2: detected capacity change from 0 to 204800000 [11366.142677] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11366.305395] LustreError: fs4241-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11366.317100] LustreError: fs4241-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11366.335450] LustreError: 308849:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11366.347946] LustreError: 308849:0:(tgt_mount.c:2129:server_put_super()) no obd fs4241-OST0001 [11366.358962] LustreError: 308849:0:(tgt_mount.c:116:server_deregister_mount()) fs4241-OST0001 not registered [11366.397471] Lustre: server umount fs4241-OST0001 complete [11366.402389] LustreError: 308849:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11375.087077] LustreError: 303888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11375.128272] LustreError: 303888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [11379.389756] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11379.594815] Lustre: fs4241-OST0001: new disk, initializing [11379.603121] Lustre: srv-fs4241-OST0001: No data found on store. Initialize space. [11379.612700] Lustre: 309401:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11379.740256] Lustre: fs4241-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11386.310333] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11395.297132] Lustre: server umount fs4241-OST0000 complete [11404.923617] Lustre: server umount fs4241-OST0001 complete [11416.617882] loop1: detected capacity change from 0 to 204800000 [11422.011458] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11422.173826] Lustre: fs4241-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11428.679351] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11436.399575] loop2: detected capacity change from 0 to 204800000 [11442.596966] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11451.660910] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11464.157591] Lustre: server umount fs4241-OST0000 complete [11470.334437] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11474.530505] Lustre: server umount fs4241-OST0001 complete [11480.558970] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11488.737931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11488.754301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11488.789103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11490.803832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11490.808616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11494.142743] Lustre: server umount lustre-OST0000 complete [11501.025747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11501.036781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11501.055251] Lustre: Skipped 1 previous similar message [11501.058385] Lustre: Skipped 1 previous similar message [11504.456660] Lustre: server umount lustre-MDT0000 complete [11506.155439] LustreError: 302049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.172500] LustreError: 302049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 69 previous similar messages [11508.897752] LustreError: 302034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852050 with bad export cookie 8128405025543462566 [11508.905746] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11508.917247] LustreError: 302034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11509.290630] Lustre: server umount lustre-MDT0001 complete [11519.511975] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11522.432113] Key type lgssc unregistered [11522.698850] LNet: 315327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11522.703256] LNetError: 315327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11522.712804] LNet: Removed LNI 192.168.201.111@tcp [11523.384347] Key type .llcrypt unregistered [11523.390420] Key type ._llcrypt unregistered [11556.712990] Key type ._llcrypt registered [11556.717374] Key type .llcrypt registered [11556.808591] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [11575.861777] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11577.330956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11577.594837] alg: No test for adler32 (adler32-zlib) [11579.105966] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [11579.551215] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11581.351221] Key type lgssc registered [11583.138834] Lustre: Echo OBD driver; http://www.lustre.org/ [11626.188933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11626.230426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11627.519176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11627.572843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11627.737984] Lustre: lustre-MDT0000: new disk, initializing [11627.852163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11627.919846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11633.104757] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11646.553442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11646.754405] Lustre: 319191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11646.832268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11646.835654] Lustre: Skipped 1 previous similar message [11646.884369] Lustre: lustre-MDT0001: new disk, initializing [11647.012624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11647.074370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11647.089665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11654.368559] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11666.889238] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11673.842237] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11682.800151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11683.085901] Lustre: lustre-OST0000: new disk, initializing [11683.090965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11683.102389] Lustre: 321161:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11683.242646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11691.103548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11691.131547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11691.244960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11692.053582] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11703.218387] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11706.239393] Lustre: Mounted lustre-client - version 2.17.57_68_geb35bb6 [11707.671558] Lustre: Unmounted lustre-client [11721.698389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11721.752147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11726.901664] Lustre: server umount lustre-OST0000 complete [11731.046292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11731.064299] Lustre: Skipped 1 previous similar message [11731.082065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11731.090822] Lustre: Skipped 1 previous similar message [11736.035424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11736.050663] Lustre: Skipped 1 previous similar message [11737.603992] Lustre: server umount lustre-MDT0000 complete [11740.959539] LustreError: 319182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852282 with bad export cookie 490544191609383479 [11740.980726] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11740.983263] LustreError: 319182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11741.318471] Lustre: server umount lustre-MDT0001 complete [11754.388333] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [11758.229287] Key type lgssc unregistered [11762.720580] LNet: 323327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11762.749807] LNetError: 323327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11834.473462] LNet: Removed LNI 192.168.201.111@tcp [11835.482158] Key type .llcrypt unregistered [11835.489057] Key type ._llcrypt unregistered [11856.288928] Key type ._llcrypt registered [11856.291398] Key type .llcrypt registered [11856.471870] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [11872.092938] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [11873.069441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11873.082171] alg: No test for adler32 (adler32-zlib) [11874.475340] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [11874.759341] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [11876.415535] Key type lgssc registered [11878.262043] Lustre: Echo OBD driver; http://www.lustre.org/ [11912.233949] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:40:51 (1787852451) [11919.668050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11919.752659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11921.200472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11921.223940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11921.336692] Lustre: lustre-MDT0000: new disk, initializing [11921.459761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11921.479905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11926.690821] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11939.268248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11939.404370] Lustre: 326553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11939.502756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11939.513754] Lustre: Skipped 1 previous similar message [11939.618632] Lustre: lustre-MDT0001: new disk, initializing [11939.728176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11939.783429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11939.806138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11944.828823] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11955.945110] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11961.803813] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11968.588537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11968.873638] Lustre: lustre-OST0000: new disk, initializing [11968.877199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11968.884205] Lustre: 328520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11968.972517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11974.951587] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [11976.257117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11976.277574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11976.440845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11983.654490] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12002.273830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12002.280749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12002.286453] LustreError: Skipped 1 previous similar message [12002.287071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12002.312223] Lustre: Skipped 1 previous similar message [12004.864801] Lustre: server umount lustre-OST0000 complete [12011.489478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12011.497707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12011.515115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12011.517450] Lustre: Skipped 1 previous similar message [12012.516096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12012.540036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12017.071943] Lustre: server umount lustre-MDT0000 complete [12022.755525] LustreError: 327715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12022.798815] LustreError: 327715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12023.625233] LustreError: 326544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852564 with bad export cookie 4426156123435267405 [12023.643742] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12024.182891] Lustre: server umount lustre-MDT0001 complete [12037.399369] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12040.490770] Key type lgssc unregistered [12040.666613] LNet: 330431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12040.670489] LNetError: 330431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12040.677622] LNet: Removed LNI 192.168.201.111@tcp [12041.521652] Key type .llcrypt unregistered [12041.524665] Key type ._llcrypt unregistered [12066.127387] Key type ._llcrypt registered [12066.138533] Key type .llcrypt registered [12066.479187] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12068.649570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12068.949657] alg: No test for adler32 (adler32-zlib) [12070.470159] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [12071.057533] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [12073.041493] Key type lgssc registered [12074.201166] Lustre: Echo OBD driver; http://www.lustre.org/ [12086.978714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12086.997727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12088.567749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12093.219490] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12098.019447] LustreError: 331706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.029516] LustreError: 331706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12100.837554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12101.081214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12105.251451] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12111.889119] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12117.822435] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12125.092328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12125.944213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12136.713811] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12147.472935] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12159.125284] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12161.038883] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12165.595662] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.11@tcp (stopping) [12167.653759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12167.669310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12167.680695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12169.187548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12169.205393] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12171.879498] Lustre: server umount lustre-MDT0000 complete [12174.309952] LustreError: 331706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12174.336520] LustreError: 331706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12175.751962] LustreError: 331703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12176.161780] LustreError: 333550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852717 with bad export cookie 5167048954636930829 [12176.169335] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12176.180848] LustreError: 333550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12179.425856] LustreError: 331702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12179.427211] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12179.473392] Lustre: Skipped 1 previous similar message [12179.483808] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12179.499506] Lustre: Skipped 3 previous similar messages [12183.090939] Lustre: server umount lustre-MDT0001 complete [12189.126466] Lustre: server umount lustre-OST0000 complete [12198.301283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12203.045766] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12206.647860] Lustre: 335889:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.111@tcp [12208.737736] Lustre: server umount MGS complete [12216.316837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12216.781039] LustreError: 336461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12216.807661] LustreError: 336461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12216.852348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12221.224196] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12225.935564] LustreError: 336457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12225.957723] LustreError: 336457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12229.589646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12229.905553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12235.055227] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12245.718032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12246.497577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12255.554430] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12264.517774] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12266.300580] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12271.760411] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12273.185450] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12278.771467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12278.789521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12278.824907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12278.829616] Lustre: Skipped 2 previous similar messages [12279.343136] Lustre: server umount lustre-OST0000 complete [12286.433476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12286.435925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12286.468718] Lustre: Skipped 1 previous similar message [12290.138275] Lustre: server umount lustre-MDT0000 complete [12291.562580] LustreError: 336462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12291.591527] LustreError: 336462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12294.275755] LustreError: 336441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852835 with bad export cookie 5167048954636931669 [12294.286816] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12294.292190] LustreError: 336441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12304.100354] Lustre: server umount lustre-MDT0001 complete [12318.094447] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12321.698803] Key type lgssc unregistered [12322.031725] LNet: 339902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12322.041941] LNetError: 339902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12322.076387] LNet: Removed LNI 192.168.201.111@tcp [12322.983806] Key type .llcrypt unregistered [12322.985858] Key type ._llcrypt unregistered [12342.137930] Key type ._llcrypt registered [12342.141254] Key type .llcrypt registered [12342.235297] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [12362.667256] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12364.217753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12364.538766] alg: No test for adler32 (adler32-zlib) [12365.787906] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [12366.262325] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [12368.135137] Key type lgssc registered [12369.708857] Lustre: Echo OBD driver; http://www.lustre.org/ [12413.360915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12413.413763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12414.778588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12414.838333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12414.951339] Lustre: lustre-MDT0000: new disk, initializing [12415.044352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12415.068081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12419.542236] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12431.215323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12431.416510] Lustre: 343064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12431.516112] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12431.520163] Lustre: Skipped 1 previous similar message [12431.629493] Lustre: lustre-MDT0001: new disk, initializing [12431.800589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12431.923893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12431.938567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12436.975796] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12446.608697] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12452.822507] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12459.155098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12459.609650] Lustre: lustre-OST0000: new disk, initializing [12459.615150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12459.622676] Lustre: 345032:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12459.726596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12465.242655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12465.255490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12465.430185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12467.145580] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12477.419146] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12482.822958] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12482.998471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12487.586647] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12487.758613] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12491.237705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12491.243396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12491.245323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12491.262519] LustreError: Skipped 1 previous similar message [12494.312732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12494.319063] Lustre: Skipped 2 previous similar messages [12495.943797] Lustre: server umount lustre-OST0000 complete [12503.520297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12503.536410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12503.552843] Lustre: Skipped 1 previous similar message [12503.557728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12504.554538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12509.670629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12509.680548] Lustre: Skipped 3 previous similar messages [12514.271354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12514.503643] Lustre: server umount lustre-MDT0000 complete [12518.715821] LustreError: 343057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853059 with bad export cookie 1101517999586031295 [12518.726266] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12518.729128] LustreError: 343057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12519.006869] Lustre: server umount lustre-MDT0001 complete [12528.368521] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:51:07 (1787853067) [12549.389776] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [12562.321357] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [12602.710205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12603.001406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12603.027473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12603.099605] Lustre: lustre-MDT0000: new disk, initializing [12603.178703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12603.194418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12608.259876] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12621.124236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12621.342862] Lustre: 349799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12621.428746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12621.434644] Lustre: Skipped 1 previous similar message [12621.541406] Lustre: lustre-MDT0001: new disk, initializing [12621.646347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12621.694728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12621.709179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12627.413666] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12639.247923] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12647.947117] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12655.917646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12656.107306] Lustre: lustre-OST0000: new disk, initializing [12656.111768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12656.131907] Lustre: 351768:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12656.238502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12662.902989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12662.922987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12663.062282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12663.495599] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [12674.543509] Lustre: DEBUG MARKER: oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12719.602160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12719.606739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12719.622730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12722.619621] Lustre: server umount lustre-OST0000 complete [12729.326259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12729.336418] LustreError: Skipped 1 previous similar message [12729.343680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12729.357844] Lustre: Skipped 1 previous similar message [12729.366685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12729.370766] Lustre: Skipped 1 previous similar message [12734.953462] LustreError: 349810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12734.990073] LustreError: 349810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12735.385922] Lustre: server umount lustre-MDT0000 complete [12739.042483] LustreError: 349809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12739.054144] LustreError: 349809:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12740.418470] LustreError: 350601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853281 with bad export cookie 1101517999586032177 [12740.425926] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12740.451128] LustreError: 350601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12741.136853] Lustre: server umount lustre-MDT0001 complete [12752.408953] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12755.738207] Key type lgssc unregistered [12756.082479] LNet: 353947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12756.092429] LNetError: 353947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12757.163640] LNet: Removed LNI 192.168.201.111@tcp [12758.258296] Key type .llcrypt unregistered [12758.260737] Key type ._llcrypt unregistered [12859.734201] Key type ._llcrypt registered [12859.737895] Key type .llcrypt registered [12859.853788] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12860.150404] Key type .llcrypt unregistered [12860.152713] Key type ._llcrypt unregistered [12941.219283] Key type ._llcrypt registered [12941.221473] Key type .llcrypt registered [12941.329455] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12941.606544] Key type .llcrypt unregistered [12941.611207] Key type ._llcrypt unregistered [12979.685661] Key type ._llcrypt registered [12979.687959] Key type .llcrypt registered [12979.762622] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [12979.974406] Key type .llcrypt unregistered [12979.976647] Key type ._llcrypt unregistered [13012.159410] Key type ._llcrypt registered [13012.162542] Key type .llcrypt registered [13012.303046] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13012.624481] Key type .llcrypt unregistered [13012.626383] Key type ._llcrypt unregistered [13096.873825] Key type ._llcrypt registered [13096.876878] Key type .llcrypt registered [13096.994564] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13097.399117] Key type .llcrypt unregistered [13097.402905] Key type ._llcrypt unregistered [13126.452769] Key type ._llcrypt registered [13126.455176] Key type .llcrypt registered [13126.561086] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13126.958197] Key type .llcrypt unregistered [13126.960055] Key type ._llcrypt unregistered [13155.936338] Key type ._llcrypt registered [13155.938760] Key type .llcrypt registered [13156.081742] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13156.423436] Key type .llcrypt unregistered [13156.429274] Key type ._llcrypt unregistered [13186.340644] Key type ._llcrypt registered [13186.346249] Key type .llcrypt registered [13186.515625] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13186.811614] Key type .llcrypt unregistered [13186.813604] Key type ._llcrypt unregistered [13218.140626] Key type ._llcrypt registered [13218.142465] Key type .llcrypt registered [13218.247903] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13218.618296] Key type .llcrypt unregistered [13218.620202] Key type ._llcrypt unregistered [13248.624675] Key type ._llcrypt registered [13248.627588] Key type .llcrypt registered [13248.752640] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13249.332438] Key type .llcrypt unregistered [13249.337866] Key type ._llcrypt unregistered [13323.379178] Key type ._llcrypt registered [13323.381764] Key type .llcrypt registered [13323.544746] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13323.957405] Key type .llcrypt unregistered [13323.961222] Key type ._llcrypt unregistered [13359.424232] Key type ._llcrypt registered [13359.425847] Key type .llcrypt registered [13359.563921] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13359.817145] Key type .llcrypt unregistered [13359.818226] Key type ._llcrypt unregistered [13387.758877] Key type ._llcrypt registered [13387.761168] Key type .llcrypt registered [13387.857243] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13388.135138] Key type .llcrypt unregistered [13388.136875] Key type ._llcrypt unregistered [13443.659414] Key type ._llcrypt registered [13443.661469] Key type .llcrypt registered [13443.737603] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [13461.520682] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13463.718490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13463.803893] alg: No test for adler32 (adler32-zlib) [13464.954881] Lustre: Lustre: Build Version: 2.17.57_68_geb35bb6 [13465.385289] LNet: Added LNI 192.168.201.111@tcp [8/256/0/180] [13467.215484] Key type lgssc registered [13468.519915] Lustre: Echo OBD driver; http://www.lustre.org/ [13505.583597] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13505.644234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13506.944106] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13506.961916] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13507.013537] Lustre: tst250fs-MDT0000: new disk, initializing [13507.074795] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13507.087119] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13511.482511] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13522.156022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13522.252882] Lustre: 374334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13522.297508] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13522.301511] Lustre: Skipped 1 previous similar message [13522.355840] Lustre: tst250fs-MDT0001: new disk, initializing [13522.447602] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13522.500518] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13522.519536] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13527.180088] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13531.887969] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13539.233130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13539.543510] Lustre: tst250fs-OST0000: new disk, initializing [13539.548646] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13539.556951] Lustre: 375968:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13539.731692] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13546.123610] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13546.144764] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13546.273188] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13546.731394] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13557.316065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13557.469074] Lustre: tst250fs-OST0001: new disk, initializing [13557.475043] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13557.486340] Lustre: 376837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13557.641869] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13563.460896] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_default_debug -1 all [13564.524609] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13564.530912] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13564.594284] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13568.504677] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13571.292809] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13571.303926] Lustre: Skipped 2 previous similar messages [13590.500602] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13590.510736] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13590.511796] Lustre: Skipped 2 previous similar messages [13590.520783] Lustre: Skipped 3 previous similar messages [13595.699831] Lustre: server umount tst250fs-MDT0000 complete [13598.716836] LustreError: 374324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854139 with bad export cookie 4635211666606796861 [13598.724849] LustreError: MGC192.168.201.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13598.727553] LustreError: 374324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13599.236207] Lustre: server umount tst250fs-MDT0001 complete [13613.498816] Lustre: server umount tst250fs-OST0000 complete [13616.353194] Lustre: 372557:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787854141/real 1787854141] req@ffff8d037a2ad680 x1874700802262912/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787854157 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13616.406577] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13616.419325] Lustre: Skipped 1 previous similar message [13617.233172] Lustre: server umount tst250fs-OST0001 complete [13636.494964] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [13647.868434] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13678.971274] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:10:18 (1787854218) [13680.191845] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13695.637782] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing set_hostid [13706.057357] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing load_modules_local [13739.389019] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13397 sec ========= 14:11:18 (1787854278) [13741.447217] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:11:20 (1787854280) === [13745.892606] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:11:24 (1787854284) === [13791.382558] Lustre: DEBUG MARKER: oleg111-server.virtnet: executing unload_modules_local [13794.068992] Key type lgssc unregistered [13794.368862] LNet: 385970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13794.375390] LNetError: 385970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13794.390901] LNet: Removed LNI 192.168.201.111@tcp [13795.125272] Key type .llcrypt unregistered [13795.127312] Key type ._llcrypt unregistered