[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 510700243 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002354] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007709] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009014] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011008] pid_max: default: 32768 minimum: 301 [ 0.012132] LSM: Security Framework initializing [ 0.013105] Yama: becoming mindful. [ 0.014039] SELinux: Initializing. [ 0.015068] *** VALIDATE selinux *** [ 0.023795] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027754] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029056] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032014] *** VALIDATE tmpfs *** [ 0.033468] *** VALIDATE proc *** [ 0.034259] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.043919] debug: unmapping init [mem 0xffffffff9ae59000-0xffffffff9ae60fff] [ 0.045166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046734] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053017] ... event mask: 000000070000000f [ 0.055257] rcu: Hierarchical SRCU implementation. [ 0.057451] smp: Bringing up secondary CPUs ... [ 0.058611] x86: Booting SMP configuration: [ 0.059028] .... node #0, CPUs: #1 #2 #3 [ 0.062547] smp: Brought up 1 node, 4 CPUs [ 0.064022] smpboot: Max logical packages: 1 [ 0.065022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.213298] node 0 deferred pages initialised in 145ms [ 0.215146] devtmpfs: initialized [ 0.216214] x86/mm: Memory block size: 128MB [ 0.218747] gcov: version magic: 0x41383552 [ 0.220307] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.221114] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.222277] pinctrl core: initialized pinctrl subsystem [ 0.223215] [ 0.223729] ************************************************************* [ 0.224013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.225018] ** ** [ 0.226021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.227015] ** ** [ 0.228016] ** This means that this kernel is built to expose internal ** [ 0.229016] ** IOMMU data structures, which may compromise security on ** [ 0.230070] ** your system. ** [ 0.231015] ** ** [ 0.232017] ** If you see this message and you are not debugging the ** [ 0.233013] ** kernel, report this immediately to your vendor! ** [ 0.234016] ** ** [ 0.235014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.236015] ************************************************************* [ 0.237674] NET: Registered protocol family 16 [ 0.238523] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.239064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.240084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.241474] cpuidle: using governor menu [ 0.244180] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.245553] PCI: Using configuration type 1 for base access [ 0.246131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.253289] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.256027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.259299] cryptd: max_cpu_qlen set to 1000 [ 0.262237] ACPI: Added _OSI(Module Device) [ 0.263012] ACPI: Added _OSI(Processor Device) [ 0.264019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.265017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.269191] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.273234] ACPI: Interpreter enabled [ 0.274106] ACPI: PM: (supports S0 S3 S4 S5) [ 0.275013] ACPI: Using IOAPIC for interrupt routing [ 0.276095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.277402] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.287498] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.288060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.289024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.290085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.292478] acpiphp: Slot [2] registered [ 0.293252] acpiphp: Slot [5] registered [ 0.294160] acpiphp: Slot [6] registered [ 0.295158] acpiphp: Slot [7] registered [ 0.296157] acpiphp: Slot [8] registered [ 0.297082] acpiphp: Slot [9] registered [ 0.298169] acpiphp: Slot [10] registered [ 0.299245] acpiphp: Slot [3] registered [ 0.300123] acpiphp: Slot [4] registered [ 0.301177] acpiphp: Slot [11] registered [ 0.302105] acpiphp: Slot [12] registered [ 0.303096] acpiphp: Slot [13] registered [ 0.304103] acpiphp: Slot [14] registered [ 0.305104] acpiphp: Slot [15] registered [ 0.306106] acpiphp: Slot [16] registered [ 0.307106] acpiphp: Slot [17] registered [ 0.308108] acpiphp: Slot [18] registered [ 0.309171] acpiphp: Slot [19] registered [ 0.310102] acpiphp: Slot [20] registered [ 0.311124] acpiphp: Slot [21] registered [ 0.312153] acpiphp: Slot [22] registered [ 0.313113] acpiphp: Slot [23] registered [ 0.314099] acpiphp: Slot [24] registered [ 0.315106] acpiphp: Slot [25] registered [ 0.316128] acpiphp: Slot [26] registered [ 0.317157] acpiphp: Slot [27] registered [ 0.318123] acpiphp: Slot [28] registered [ 0.319106] acpiphp: Slot [29] registered [ 0.320106] acpiphp: Slot [30] registered [ 0.321106] acpiphp: Slot [31] registered [ 0.322067] PCI host bridge to bus 0000:00 [ 0.323026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.326026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.328026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.331068] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.333493] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339732] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.342484] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.344018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346541] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347819] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.348038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.349780] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.351756] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.358018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.360985] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.364609] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.367020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.370018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.377021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.384128] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.387018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.390021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.398019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.404651] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.407016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.410025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.417018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.423705] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.426020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.429022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.436018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.444673] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.455024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.462022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.477024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.488018] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.497022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.505025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.529022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.540115] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.542382] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.544402] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.547667] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.550263] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.556041] iommu: Default domain type: Passthrough [ 0.558352] SCSI subsystem initialized [ 0.559157] ACPI: bus type USB registered [ 0.561099] usbcore: registered new interface driver usbfs [ 0.562072] usbcore: registered new interface driver hub [ 0.564102] usbcore: registered new device driver usb [ 0.566144] pps_core: LinuxPPS API ver. 1 registered [ 0.567013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.570062] PTP clock support registered [ 0.572107] EDAC MC: Ver: 3.0.0 [ 0.581585] PCI: Using ACPI for IRQ routing [ 0.584083] NetLabel: Initializing [ 0.586014] NetLabel: domain hash size = 128 [ 0.588016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.590097] NetLabel: unlabeled traffic allowed by default [ 0.592120] vgaarb: loaded [ 0.593275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.595019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.600004] clocksource: Switched to clocksource kvm-clock [ 0.719347] VFS: Disk quotas dquot_6.6.0 [ 0.720730] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.723039] *** VALIDATE ramfs *** [ 0.724188] *** VALIDATE hugetlbfs *** [ 0.725735] pnp: PnP ACPI init [ 0.728072] pnp: PnP ACPI: found 6 devices [ 0.744351] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.747293] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.749192] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.751072] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.753286] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.755321] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.757945] NET: Registered protocol family 2 [ 0.760069] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.764494] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.767273] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.772178] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.775792] TCP: Hash tables configured (established 65536 bind 65536) [ 0.777796] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.780105] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.782669] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.785883] NET: Registered protocol family 1 [ 0.789307] RPC: Registered named UNIX socket transport module. [ 0.791487] RPC: Registered udp transport module. [ 0.793021] RPC: Registered tcp transport module. [ 0.794533] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.796553] NET: Registered protocol family 44 [ 0.798077] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.801289] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.803667] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.806133] PCI: CLS 0 bytes, default 64 [ 0.808201] Unpacking initramfs... [ 2.267830] debug: unmapping init [mem 0xffffa0fe7cc54000-0xffffa0fe7ffbffff] [ 2.271808] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.274165] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.277094] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.806773] Initialise system trusted keyrings [ 2.808603] Key type blacklist registered [ 2.812064] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.823128] zbud: loaded [ 2.826355] *** VALIDATE nfs *** [ 2.827593] *** VALIDATE nfs4 *** [ 2.829227] pstore: using deflate compression [ 2.832086] Platform Keyring initialized [ 2.936922] NET: Registered protocol family 38 [ 2.938624] Key type asymmetric registered [ 2.940018] Asymmetric key parser 'x509' registered [ 2.941636] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.944489] io scheduler mq-deadline registered [ 2.945754] io scheduler kyber registered [ 2.947076] io scheduler bfq registered [ 2.948681] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.951402] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.953899] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.956248] ACPI: Power Button [PWRF] [ 2.963149] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.969775] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.983172] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.990165] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.002987] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.031297] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.059887] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.064904] Non-volatile memory driver v1.3 [ 3.066344] Linux agpgart interface v0.103 [ 3.098958] virtio_blk virtio1: [vda] 146008 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.102631] vda: detected capacity change from 0 to 74756096 [ 3.115628] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.117903] vdb: detected capacity change from 0 to 1073741824 [ 3.131996] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.135555] vdc: detected capacity change from 0 to 2621440000 [ 3.153489] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.156091] vdd: detected capacity change from 0 to 2621440000 [ 3.173525] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.176166] vde: detected capacity change from 0 to 4294967296 [ 3.195954] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.199392] vdf: detected capacity change from 0 to 4294967296 [ 3.206311] libphy: Fixed MDIO Bus: probed [ 3.212550] usbcore: registered new interface driver usbserial_generic [ 3.215575] usbserial: USB Serial support registered for generic [ 3.218039] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.222467] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.224909] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.227173] mousedev: PS/2 mouse device common for all mice [ 3.229673] rtc_cmos 00:05: RTC can wake from S4 [ 3.232160] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.232898] rtc_cmos 00:05: registered as rtc0 [ 3.237337] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.237756] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.241988] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.243607] intel_pstate: CPU model not supported [ 3.249260] hid: raw HID events driver (C) Jiri Kosina [ 3.251766] usbcore: registered new interface driver usbhid [ 3.254365] usbhid: USB HID core driver [ 3.256189] drop_monitor: Initializing network drop monitor service [ 3.259067] Initializing XFRM netlink socket [ 3.261374] NET: Registered protocol family 10 [ 3.264911] Segment Routing with IPv6 [ 3.266756] NET: Registered protocol family 17 [ 3.269068] mpls_gso: MPLS GSO support [ 3.274819] RAS: Correctable Errors collector initialized. [ 3.276945] AVX version of gcm_enc/dec engaged. [ 3.278287] AES CTR mode by8 optimization enabled [ 3.349625] sched_clock: Marking stable (3349601609, 0)->(4486110214, -1136508605) [ 3.353155] registered taskstats version 1 [ 3.355433] Loading compiled-in X.509 certificates [ 3.357422] zswap: loaded using pool lzo/zbud [ 3.385163] Key type big_key registered [ 3.397887] Key type encrypted registered [ 3.399875] ima: No TPM chip found, activating TPM-bypass! [ 3.402196] ima: Allocated hash algorithm: sha1 [ 3.404140] ima: No architecture policies found [ 3.406662] evm: Initialising EVM extended attributes: [ 3.408948] evm: security.selinux [ 3.410504] evm: security.ima [ 3.411787] evm: security.capability [ 3.413394] evm: HMAC attrs: 0x1 [ 3.416120] rtc_cmos 00:05: setting system clock to 2026-08-22 00:55:39 UTC (1787360139) [ 3.423288] debug: unmapping init [mem 0xffffffff9be03000-0xffffffff9bffffff] [ 3.427051] debug: unmapping init [mem 0xffffffff9ab82000-0xffffffff9ae58fff] [ 3.434122] Write protecting the kernel read-only data: 28672k [ 3.438490] debug: unmapping init [mem 0xffffffff99203000-0xffffffff993fffff] [ 3.441829] debug: unmapping init [mem 0xffffffff99b14000-0xffffffff99bfffff] [ 3.478726] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.491731] systemd[1]: Detected virtualization kvm. [ 3.493567] systemd[1]: Detected architecture x86-64. [ 3.495362] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.521721] systemd[1]: No hostname configured. [ 3.523853] systemd[1]: Set hostname to . [ 3.526059] random: systemd: uninitialized urandom read (16 bytes read) [ 3.528384] systemd[1]: Initializing machine ID from random generator. [ 3.597750] random: ln: uninitialized urandom read (6 bytes read) [ 3.703083] random: systemd: uninitialized urandom read (16 bytes read) [ 3.705841] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.709656] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.714762] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.329191] device-mapper: uevent: version 1.0.3 [ 4.331509] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.121149] virtio_net virtio0 ens2: renamed from eth0 [ 5.144089] random: fast init done [ 5.194547] scsi host0: ata_piix [ 5.217545] scsi host1: ata_piix [ 5.219257] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.221593] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.776124] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.919232] random: crng init done [ 9.922095] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.492410] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.746939] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.017953] SELinux: Disabled at runtime. [ 12.087559] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.095553] systemd[1]: Detected virtualization kvm. [ 12.097418] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.580608] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.583923] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.589091] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.592875] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.596290] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.603069] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.612224] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ 12.757751] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.043575] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.334703] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.344133] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.543256] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.569390] EDAC sbridge: Ver: 1.1.2 [ 15.236167] Key type dns_resolver registered [ 15.548918] NFS: Registering the id_resolver key type [ 15.551235] Key type id_resolver registered [ 15.552790] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg232-server login: [ 45.925257] libcfs: loading out-of-tree module taints kernel. [ 45.968767] Key type ._llcrypt registered [ 45.972279] Key type .llcrypt registered [ 46.092046] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 70.383801] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 72.657533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.666876] alg: No test for adler32 (adler32-zlib) [ 74.530718] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 75.738542] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 77.551486] Key type lgssc registered [ 79.780956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 101.976419] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 152.401958] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 156.984877] hrtimer: interrupt took 4885957 ns [ 168.742659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 168.794430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.188669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 170.232366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 170.385293] Lustre: lustre-MDT0000: new disk, initializing [ 170.478175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.496583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 174.970909] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.811149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.947343] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 189.976702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 189.981294] Lustre: Skipped 1 previous similar message [ 190.108760] Lustre: lustre-MDT0001: new disk, initializing [ 190.191318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.215557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.229063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 194.974420] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.665489] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 212.359312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.671971] Lustre: lustre-OST0000: new disk, initializing [ 212.676294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 212.683664] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.794480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.639575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 213.646722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 213.797521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.443979] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.819861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.995688] Lustre: lustre-OST0001: new disk, initializing [ 236.998985] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.007548] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.103651] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.280503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 242.290521] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 242.374354] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 245.036759] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.607339] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 269.179129] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.075747] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.732958] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing yml_node [ 290.057369] Lustre: DEBUG MARKER: Client: 2.17.57.45 [ 293.581755] Lustre: DEBUG MARKER: MDS: 2.17.57.45 [ 296.052834] Lustre: DEBUG MARKER: OSS: 2.17.57.45 [ 298.034859] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 21 21:00:31 EDT 2026 [ 318.484345] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 320.156730] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.914551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 338.926513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.957378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.936285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.944420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.878884] Lustre: server umount lustre-MDT0000 complete [ 347.330088] LustreError: 7458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787360483 with bad export cookie 1439340629393217884 [ 347.343700] LustreError: 7458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 347.346223] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 350.175752] LustreError: 8928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.180792] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.188626] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.188648] Lustre: Skipped 3 previous similar messages [ 350.191260] LustreError: 8928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 350.209782] Lustre: Skipped 3 previous similar messages [ 352.224917] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.225566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.264906] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 352.277086] Lustre: Skipped 2 previous similar messages [ 354.052748] Lustre: server umount lustre-MDT0001 complete [ 358.813866] Lustre: server umount lustre-OST0000 complete [ 363.009755] Lustre: server umount lustre-OST0001 complete [ 370.443497] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 382.822302] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 424.939545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.225617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 425.266505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 425.442909] Lustre: lustre-MDT0000: new disk, initializing [ 425.562489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.580571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 430.679435] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 442.382625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.511475] Lustre: 15091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 442.580449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 442.587056] Lustre: Skipped 1 previous similar message [ 442.673922] Lustre: lustre-MDT0001: new disk, initializing [ 442.759376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 442.797076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 442.813751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 447.585132] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 457.971176] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 464.121802] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 470.816256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 471.179674] Lustre: lustre-OST0000: new disk, initializing [ 471.190469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 471.201803] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 471.356864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.459824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 472.467748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 472.621704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 477.943578] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 488.066365] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 493.450285] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 493.710997] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 499.705899] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 499.998565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 503.290119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.301729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 503.312344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.327294] Lustre: Skipped 1 previous similar message [ 508.665479] Lustre: server umount lustre-OST0000 complete [ 513.509275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.521949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 513.528185] Lustre: Skipped 2 previous similar messages [ 513.550301] Lustre: Skipped 3 previous similar messages [ 519.549302] Lustre: server umount lustre-MDT0000 complete [ 523.744396] LustreError: 15103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.768086] LustreError: 15103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 523.818988] LustreError: 15084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787360659 with bad export cookie 1439340629393219417 [ 523.828726] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.832891] LustreError: 15084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 524.139523] Lustre: server umount lustre-MDT0001 complete [ 531.024553] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:04:23 (1787360663) [ 553.035852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.545216] LustreError: 19939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.664156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 558.156363] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 559.077898] LustreError: 19940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.861642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.971542] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 580.835352] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 588.064727] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 590.447947] Lustre: Failing over lustre-MDT0000 [ 590.751914] Lustre: server umount lustre-MDT0000 complete [ 593.569616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.574072] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.593209] Lustre: Skipped 1 previous similar message [ 593.612796] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 609.744132] Lustre: 3640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787360729/real 1787360729] req@ffffa0fdc334ca80 x1874183020776960/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787360745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 609.763807] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.870743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.865442] LustreError: 19940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.883300] LustreError: 19940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 619.316126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.322788] Lustre: Skipped 1 previous similar message [ 619.382755] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 624.374445] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 624.610843] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 624.623805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 624.670340] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 635.192406] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 639.968824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 639.981325] LustreError: Skipped 1 previous similar message [ 639.988171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.009874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.013893] Lustre: Skipped 2 previous similar messages [ 654.303182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 654.587312] Lustre: server umount lustre-MDT0000 complete [ 658.404307] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.424989] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 658.878432] LustreError: 19921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787360794 with bad export cookie 1439340629393220831 [ 658.884171] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 658.895145] LustreError: 19921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 659.409848] Lustre: server umount lustre-MDT0001 complete [ 669.736184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.190161] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 685.692680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.472204] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 700.515426] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 708.051740] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 710.489112] Lustre: Failing over lustre-MDT0000 [ 710.514550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.524572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 710.526280] Lustre: Skipped 2 previous similar messages [ 710.536640] Lustre: Skipped 7 previous similar messages [ 711.002479] Lustre: server umount lustre-MDT0000 complete [ 727.009917] LustreError: 23283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 727.012139] Lustre: 3640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787360847/real 1787360847] req@ffffa0ff01b73800 x1874183020807808/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787360863 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 727.032083] LustreError: 23283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 727.072193] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 731.927115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 738.273421] LustreError: 3639:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa0ff01b74700 x1874183020810496/t0(0) o250->MGC192.168.202.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 738.638312] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 743.925747] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 743.952991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 743.972952] Lustre: Skipped 1 previous similar message [ 744.007329] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 744.062758] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 754.958244] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 759.266082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 765.605697] Lustre: server umount lustre-MDT0000 complete [ 769.412946] LustreError: 24855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787360905 with bad export cookie 1439340629393221769 [ 769.413693] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.431879] LustreError: 24855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 770.049918] Lustre: server umount lustre-MDT0001 complete [ 781.041511] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:08:33 (1787360913) [ 801.797746] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 816.530317] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 866.350288] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 880.323063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.779415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 880.822927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 881.013495] Lustre: lustre-MDT0000: new disk, initializing [ 881.127326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.139238] Lustre: Skipped 3 previous similar messages [ 881.181065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 886.106511] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 898.252912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.404720] Lustre: 29682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 898.628682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 904.251253] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 916.851607] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 924.103477] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 933.467136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 933.784318] Lustre: 31656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 935.850730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 935.938449] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 936.028041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 936.044734] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 942.271647] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 953.635456] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 960.917480] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 987.125470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 987.130278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.137065] LustreError: Skipped 1 previous similar message [ 987.145222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 987.160499] Lustre: Skipped 3 previous similar messages [ 987.175408] Lustre: Skipped 4 previous similar messages [ 996.339895] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 996.595871] Lustre: server umount lustre-OST0000 complete [ 1007.385408] Lustre: server umount lustre-MDT0000 complete [ 1007.595456] LustreError: 29690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.629851] LustreError: 29690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 1011.810672] LustreError: 29676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787361147 with bad export cookie 1439340629393222168 [ 1011.823174] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.485470] Lustre: server umount lustre-MDT0001 complete [ 1025.137797] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1027.793416] Key type lgssc unregistered [ 1028.063071] LNet: 33639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.073373] LNetError: 33639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.083471] LNet: Removed LNI 192.168.202.132@tcp [ 1028.998155] Key type .llcrypt unregistered [ 1029.000105] Key type ._llcrypt unregistered [ 1054.601906] Key type ._llcrypt registered [ 1054.603610] Key type .llcrypt registered [ 1054.747186] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1056.019967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1056.047196] alg: No test for adler32 (adler32-zlib) [ 1057.121948] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1057.322878] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1058.975235] Key type lgssc registered [ 1060.737546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1071.162357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1071.172280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.695541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1077.300978] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1082.849484] LustreError: 35053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.869808] LustreError: 35053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1086.051267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.391557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.231566] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1099.964805] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1106.388330] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1114.164342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.557140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.639880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1121.701874] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1130.292683] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1131.963924] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1145.941590] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:14:39 (1787361279) [ 1148.406742] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1150.461329] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1160.317677] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:14:52 (1787361292) [ 1171.579500] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:15:04 (1787361304) [ 1173.600872] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1185.414793] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1185.422989] Lustre: Skipped 2 previous similar messages [ 1187.038366] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:15:20 (1787361320) [ 1191.120382] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1191.124405] Lustre: Skipped 3 previous similar messages [ 1210.469146] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1210.476784] Lustre: Skipped 6 previous similar messages [ 1213.735410] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:15:46 (1787361346) [ 1216.582419] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1220.075544] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1229.762979] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:16:03 (1787361363) [ 1231.917657] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1231.920694] Lustre: Skipped 3 previous similar messages [ 1248.350565] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:16:21 (1787361381) [ 1268.199655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.203150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1268.216166] Lustre: Skipped 1 previous similar message [ 1273.318982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1273.335959] Lustre: Skipped 2 previous similar messages [ 1274.262293] Lustre: server umount lustre-OST0000 complete [ 1280.992427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1281.004180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.021652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1283.554533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.563404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1283.608447] Lustre: Skipped 1 previous similar message [ 1285.110947] Lustre: server umount lustre-MDT0000 complete [ 1288.686981] LustreError: 36292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1288.716485] LustreError: 36292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1289.867308] LustreError: 35032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787361425 with bad export cookie 11660458589604946652 [ 1289.879480] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1289.897275] LustreError: 35032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1290.360631] Lustre: server umount lustre-MDT0001 complete [ 1306.623181] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1310.445972] Key type lgssc unregistered [ 1310.803534] LNet: 42922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.809374] LNetError: 42922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.846809] LNet: Removed LNI 192.168.202.132@tcp [ 1312.899665] Key type .llcrypt unregistered [ 1312.903818] Key type ._llcrypt unregistered [ 1337.196203] Key type ._llcrypt registered [ 1337.198888] Key type .llcrypt registered [ 1337.302811] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 1359.101725] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1360.265297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1360.314593] alg: No test for adler32 (adler32-zlib) [ 1361.581282] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1361.837230] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1363.463162] Key type lgssc registered [ 1365.011785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1405.567796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1405.602261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.958200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1406.988765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1407.160408] Lustre: lustre-MDT0000: new disk, initializing [ 1407.308575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1407.326654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1412.464657] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1425.024639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.190953] Lustre: 46084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1425.261953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1425.265982] Lustre: Skipped 1 previous similar message [ 1425.361773] Lustre: lustre-MDT0001: new disk, initializing [ 1425.553249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1425.615156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1425.639347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1430.563723] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1440.836788] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1448.447241] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1457.249287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1457.708342] Lustre: lustre-OST0000: new disk, initializing [ 1457.711669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1457.717920] Lustre: 48051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1457.830602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.437138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1464.460735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1464.613467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1464.635532] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1476.070865] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1482.806417] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1483.110617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1489.463161] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1489.765929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1495.522202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1495.522414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.529924] LustreError: Skipped 1 previous similar message [ 1495.540461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1495.548047] Lustre: Skipped 1 previous similar message [ 1495.577146] Lustre: Skipped 1 previous similar message [ 1498.008257] Lustre: server umount lustre-OST0000 complete [ 1505.767066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1505.773645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1505.776831] Lustre: Skipped 1 previous similar message [ 1508.999136] Lustre: server umount lustre-MDT0000 complete [ 1513.704806] LustreError: 46076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787361649 with bad export cookie 2613711912631451805 [ 1513.710774] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1513.745707] LustreError: 46076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1514.164661] Lustre: server umount lustre-MDT0001 complete [ 1525.430718] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 21:20:58 (1787361658) [ 1533.837556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.435173] LustreError: 50511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.455231] LustreError: 50511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1534.499737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.396565] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1539.555851] LustreError: 50512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.672278] LustreError: 50511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.666341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.157825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1553.858340] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1562.418241] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1568.853145] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1575.569353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1575.857363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.063530] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1590.628270] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1598.633413] LustreError: 53041:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1599.444941] Lustre: Modifying parameter general.timeout=1 in log params [ 1601.136802] Lustre: Modifying parameter general.timeout=3 in log params [ 1601.141467] Lustre: Skipped 1 previous similar message [ 1603.193763] Lustre: Modifying parameter general.timeout=5 in log params [ 1603.198177] Lustre: Skipped 1 previous similar message [ 1608.176314] Lustre: Modifying parameter general.timeout=10 in log params [ 1608.182537] Lustre: Skipped 4 previous similar messages [ 1616.841149] Lustre: Modifying parameter general.timeout=19 in log params [ 1616.843743] Lustre: Skipped 8 previous similar messages [ 1633.325563] Lustre: Modifying parameter general.timeout=37 in log params [ 1633.329166] Lustre: Skipped 17 previous similar messages [ 1665.846954] Lustre: Modifying parameter general.timeout=66 in log params [ 1665.855776] Lustre: Skipped 28 previous similar messages [ 1738.108181] Lustre: Disabling parameter general.timeout= in log params [ 1738.111928] Lustre: Skipped 63 previous similar messages [ 1741.760630] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 21:24:34 (1787361874) [ 1745.083321] Lustre: 60005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1747.235225] Lustre: 60103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1750.769633] Lustre: 60300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1750.779259] Lustre: 60300:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1755.077653] Lustre: 60510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1755.086859] Lustre: 60510:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1787.913308] Lustre: 61873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1787.938134] LustreError: 61873:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1793.712427] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 21:25:26 (1787361926) [ 1808.868379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.869818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1808.897227] Lustre: Skipped 1 previous similar message [ 1808.906309] Lustre: Skipped 1 previous similar message [ 1813.098084] Lustre: server umount lustre-MDT0000 complete [ 1813.984495] LustreError: 51520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1814.002267] LustreError: 51520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1816.854703] LustreError: 50494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787361952 with bad export cookie 2613711912631452687 [ 1816.858066] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1816.872612] LustreError: 50494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1817.235573] Lustre: server umount lustre-MDT0001 complete [ 1831.667187] Lustre: server umount lustre-OST0000 complete [ 1863.743760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.027604] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1864.263912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.149748] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1884.051768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1884.315910] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1884.341318] Lustre: 64406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1890.503321] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1899.954497] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1907.133530] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1915.869395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1916.158181] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1916.592760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1916.600324] Lustre: Skipped 1 previous similar message [ 1923.931467] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1934.081292] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1936.254593] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1936.274769] Lustre: Skipped 23 previous similar messages [ 1936.352561] Lustre: 66973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1945.571434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1945.585154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.598235] Lustre: Skipped 1 previous similar message [ 1945.608302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1945.615019] Lustre: Skipped 2 previous similar messages [ 1950.180477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1950.189399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.214060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1951.582436] Lustre: server umount lustre-OST0000 complete [ 1960.934218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.963783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.968523] Lustre: Skipped 2 previous similar messages [ 1963.573027] Lustre: server umount lustre-MDT0000 complete [ 1966.049192] LustreError: 64416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.066048] LustreError: 64416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1968.783478] LustreError: 64397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787362104 with bad export cookie 2613711912631455900 [ 1968.784989] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.802689] LustreError: 64397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1969.708653] Lustre: server umount lustre-MDT0001 complete [ 1984.269745] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1987.303837] Key type lgssc unregistered [ 1987.583879] LNet: 68188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1987.594561] LNetError: 68188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1987.619640] LNet: Removed LNI 192.168.202.132@tcp [ 1988.381869] Key type .llcrypt unregistered [ 1988.386371] Key type ._llcrypt unregistered [ 2017.229335] Key type ._llcrypt registered [ 2017.231927] Key type .llcrypt registered [ 2017.385675] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2018.918623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2018.947227] alg: No test for adler32 (adler32-zlib) [ 2020.192613] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2020.606340] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2022.319204] Key type lgssc registered [ 2024.253651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2033.922647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2033.956641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.607147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.628101] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2045.928912] LustreError: 69606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.954222] LustreError: 69606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2051.044891] LustreError: 69607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.722896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2052.326934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2059.395818] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2069.374318] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2077.878265] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2086.907455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2087.670316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.278150] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2102.211408] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2116.115582] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2128.869870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2128.875112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2128.891716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.942466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.943052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.953994] Lustre: Skipped 1 previous similar message [ 2131.963779] Lustre: Skipped 1 previous similar message [ 2133.827678] Lustre: server umount lustre-MDT0000 complete [ 2137.066053] LustreError: 69603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.094793] LustreError: 69603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2138.317257] LustreError: 69585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787362274 with bad export cookie 5139244842645527551 [ 2138.320847] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2138.329077] LustreError: 69585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2138.902526] Lustre: server umount lustre-MDT0001 complete [ 2153.941240] Lustre: server umount lustre-OST0000 complete [ 2182.948316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.179930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2183.215226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2183.613729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.470739] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2197.454788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.552447] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2197.574558] Lustre: 74478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2197.930392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.687734] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2215.558520] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2223.105339] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2230.939965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2231.179458] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2231.503326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.250043] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2251.333289] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2254.610508] Lustre: 77054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2256.595861] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2261.985625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2261.999651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2262.046231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2262.054628] Lustre: Skipped 1 previous similar message [ 2267.619085] LustreError: 76331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.702796] Lustre: server umount lustre-OST0000 complete [ 2272.748091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.752549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2272.763340] Lustre: Skipped 2 previous similar messages [ 2272.788638] Lustre: Skipped 3 previous similar messages [ 2277.860758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2277.865368] Lustre: Skipped 1 previous similar message [ 2286.047293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2286.327477] Lustre: server umount lustre-MDT0000 complete [ 2288.103248] LustreError: 74484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.126847] LustreError: 74484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2290.551775] LustreError: 74471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787362426 with bad export cookie 5139244842645528398 [ 2290.560765] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2290.584711] LustreError: 74471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2291.081190] Lustre: server umount lustre-MDT0001 complete [ 2304.408340] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2307.581796] Key type lgssc unregistered [ 2307.941661] LNet: 78269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.947913] LNetError: 78269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2307.983084] LNet: Removed LNI 192.168.202.132@tcp [ 2309.052442] Key type .llcrypt unregistered [ 2309.055952] Key type ._llcrypt unregistered [ 2336.092865] Key type ._llcrypt registered [ 2336.095769] Key type .llcrypt registered [ 2336.234596] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2337.416564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2337.542345] alg: No test for adler32 (adler32-zlib) [ 2338.579866] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2338.864731] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2340.599179] Key type lgssc registered [ 2342.125473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2350.414306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2350.437592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2352.088270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2357.917084] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2362.338205] LustreError: 79686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.359846] LustreError: 79686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2367.466856] LustreError: 79687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.731290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.303255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2375.809884] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2386.740699] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2394.144865] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2402.255865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2402.927930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.082838] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2420.674792] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2431.152719] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2432.021035] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2433.347415] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2435.455192] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2435.457993] Lustre: Skipped 1 previous similar message [ 2439.660808] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2439.678781] Lustre: Skipped 3 previous similar messages [ 2447.953834] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2447.956805] Lustre: Skipped 7 previous similar messages [ 2464.084870] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2464.091374] Lustre: Skipped 16 previous similar messages [ 2496.993170] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2496.996126] Lustre: Skipped 31 previous similar messages [ 2545.626853] Lustre: 87791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2550.595605] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:38:02 (1787362682) [ 2553.454629] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2556.082456] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:38:08 (1787362688) [ 2563.054344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2563.071486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.108472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2566.121430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.129177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2566.149535] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2566.163741] Lustre: Skipped 1 previous similar message [ 2569.056946] Lustre: server umount lustre-OST0000 complete [ 2576.353780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.362497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2576.387758] Lustre: Skipped 1 previous similar message [ 2576.398660] Lustre: Skipped 1 previous similar message [ 2581.473220] LustreError: 79687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.925199] Lustre: server umount lustre-MDT0000 complete [ 2586.512224] LustreError: 81538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787362722 with bad export cookie 10779682677924151897 [ 2586.520504] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2586.523337] LustreError: 81538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2587.044256] Lustre: server umount lustre-MDT0001 complete [ 2600.974921] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2603.929293] Key type lgssc unregistered [ 2604.220761] LNet: 89044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2604.238664] LNetError: 89044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2604.269091] LNet: Removed LNI 192.168.202.132@tcp [ 2605.048633] Key type .llcrypt unregistered [ 2605.050714] Key type ._llcrypt unregistered [ 2610.172678] Key type ._llcrypt registered [ 2610.174596] Key type .llcrypt registered [ 2610.312511] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2615.189721] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules [ 2616.793705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2616.864810] alg: No test for adler32 (adler32-zlib) [ 2617.902934] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2618.115505] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2619.792154] Key type lgssc registered [ 2620.882873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2621.886834] LustreError: 90246:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2639.883795] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2644.347187] LustreError: 90246:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2650.398362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2650.411550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.965201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.500730] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2661.344608] LustreError: 91272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.369625] LustreError: 91272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2666.467959] LustreError: 91273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.854986] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:40:01 (1787362801) [ 2671.595467] LustreError: 91272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.941566] LustreError: 91304:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2674.969907] LustreError: 91304:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2674.992342] LustreError: 91304:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2676.028315] Lustre: server umount lustre-MDT0000 complete [ 2696.345569] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2700.510991] Key type lgssc unregistered [ 2700.923963] LNet: 92802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2700.948188] LNetError: 92802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2700.977276] LNet: Removed LNI 192.168.202.132@tcp [ 2702.330912] Key type .llcrypt unregistered [ 2702.338152] Key type ._llcrypt unregistered [ 2725.760600] Key type ._llcrypt registered [ 2725.770344] Key type .llcrypt registered [ 2725.937657] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2727.844073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2727.939298] alg: No test for adler32 (adler32-zlib) [ 2729.019332] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2729.286651] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2730.975192] Key type lgssc registered [ 2731.789179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2741.687643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2741.724954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.456278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2748.566928] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2752.996032] LustreError: 94029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.018156] LustreError: 94029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2758.114544] LustreError: 94030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.145137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.280848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2764.607896] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2773.018669] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2779.788352] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2787.727502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.712635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2796.259825] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2804.319805] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2861.798507] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 21:43:14 (1787362994) [ 2864.811562] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2867.041610] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 21:43:20 (1787363000) [ 2874.851291] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2874.854254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2874.880452] Lustre: Skipped 1 previous similar message [ 2879.957399] Lustre: server umount lustre-MDT0000 complete [ 2879.974657] LustreError: 94030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.981640] LustreError: 94030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2884.834664] LustreError: 94011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787363020 with bad export cookie 15521887712232694983 [ 2884.850777] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2884.860162] LustreError: 94011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2885.095334] LustreError: 94024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.098614] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.143515] Lustre: Skipped 1 previous similar message [ 2885.153788] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2885.158188] Lustre: Skipped 2 previous similar messages [ 2885.358504] Lustre: server umount lustre-MDT0001 complete [ 2892.856877] Lustre: server umount lustre-OST0000 complete [ 2906.690375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.016962] LustreError: 98326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.228496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.078789] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2917.349144] LustreError: 98327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2917.375739] LustreError: 98327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2921.966526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.049801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2929.261433] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2940.287696] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2947.330558] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2963.241910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2963.461117] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2963.475025] LustreError: 98316:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2963.498894] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2963.507766] LustreError: 100414:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2963.516319] LustreError: 100414:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2963.524650] LustreError: 100414:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2963.626460] Lustre: server umount lustre-OST0000 complete [ 2963.631650] LustreError: 100414:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2972.758755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2973.098506] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2973.104829] LustreError: 98314:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2973.131962] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2973.141985] LustreError: 100880:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2973.147959] LustreError: 100880:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2973.152448] LustreError: 100880:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2973.205671] Lustre: server umount lustre-OST0000 complete [ 2973.208226] LustreError: 100880:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2983.311288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2983.570269] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2983.574213] Lustre: Found index 0 for lustre-OST0000, updating log [ 2983.578517] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2983.633860] Lustre: lustre-OST0000: new disk, initializing [ 2983.637653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2983.643236] Lustre: 101414:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2983.881608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.450906] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2995.448428] Lustre: Failing over lustre-OST0000 [ 2995.714534] Lustre: server umount lustre-OST0000 complete [ 3002.338298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.340292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3002.348510] Lustre: Skipped 1 previous similar message [ 3002.375965] Lustre: Skipped 1 previous similar message [ 3006.283155] Lustre: server umount lustre-MDT0000 complete [ 3007.457154] LustreError: 98326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.475085] LustreError: 98326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3010.404825] LustreError: 98354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787363146 with bad export cookie 15521887712232697727 [ 3010.406169] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3010.414351] LustreError: 98354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3020.980884] Lustre: server umount lustre-MDT0001 complete [ 3030.767454] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 21:46:03 (1787363163) [ 3050.379318] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3063.044105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.475223] LustreError: 103762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.647846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3069.336706] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3080.512978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.508223] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3091.010475] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3099.659844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3100.520693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3100.535551] Lustre: Skipped 1 previous similar message [ 3100.553848] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3107.684361] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3107.727878] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3107.728633] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3108.156620] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3117.996722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3118.166018] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3118.352916] Lustre: lustre-OST0001: new disk, initializing [ 3118.361042] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3118.378643] Lustre: 106086:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3125.676737] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3127.893904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3127.907089] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3128.014397] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3134.146576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3140.283133] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3148.271736] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3148.294834] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.321762] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3150.863923] Lustre: server umount lustre-MDT0001 complete [ 3153.892486] LustreError: 104928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.914247] LustreError: 104928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3160.622991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.783768] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3160.788627] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3160.863848] Lustre: 103752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3162.711179] LustreError: 107629:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3162.719793] LustreError: 107629:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3162.742515] LustreError: 103754:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3162.923680] LustreError: 107643:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.132@tcp: cfg command failed: rc = -17 [ 3162.933659] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3162.933659] [ 3167.234983] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3167.428596] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3180.149200] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 21:48:32 (1787363312) [ 3200.480311] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3220.745847] Lustre: 109700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3246.960940] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3250.595631] Lustre: 110835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3340.770397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3340.787466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.811633] Lustre: Skipped 2 previous similar messages [ 3340.843622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3340.854550] Lustre: Skipped 3 previous similar messages [ 3342.717329] Lustre: server umount lustre-MDT0000 complete [ 3346.401276] LustreError: 104928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3346.411632] LustreError: 104928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3347.321313] LustreError: 110837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787363483 with bad export cookie 15521887712232698728 [ 3347.333298] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3347.343350] LustreError: 110837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3347.863586] Lustre: server umount lustre-MDT0001 complete [ 3362.409540] Lustre: server umount lustre-OST0000 complete [ 3377.164217] Lustre: server umount lustre-OST0001 complete [ 3382.639505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3391.793963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3402.070165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3414.651803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3436.582422] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3451.398451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.420157] Lustre: lustre-MDT0000: reset Object Index mappings [ 3451.985092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3451.993793] Lustre: Skipped 2 previous similar messages [ 3456.553089] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3465.676082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3465.710141] Lustre: lustre-MDT0001: reset Object Index mappings [ 3472.451368] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3475.960298] Lustre: 115942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3483.472371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3491.174786] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3491.340085] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3496.453846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3500.403896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3506.685270] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3506.701201] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3507.374718] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3514.754198] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3518.580982] Lustre: 117818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3571.696576] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 21:55:04 (1787363704) [ 3598.820366] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.834071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.839214] Lustre: Skipped 6 previous similar messages [ 3598.861656] Lustre: Skipped 7 previous similar messages [ 3602.906281] Lustre: server umount lustre-MDT0000 complete [ 3603.949322] LustreError: 114789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3603.983372] LustreError: 114789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3606.869188] LustreError: 117821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787363742 with bad export cookie 15521887712232745642 [ 3606.878554] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3606.888260] LustreError: 117821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3616.243554] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3616.258441] Lustre: Skipped 5 previous similar messages [ 3621.346913] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3622.234518] Lustre: server umount lustre-MDT0001 complete [ 3634.465911] Lustre: server umount lustre-OST0000 complete [ 3647.296568] Lustre: server umount lustre-OST0001 complete [ 3655.485606] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 3667.755352] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3705.835857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.044409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3706.076446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3706.147488] Lustre: lustre-MDT0000: new disk, initializing [ 3706.212213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3706.218023] Lustre: Skipped 3 previous similar messages [ 3706.235168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3711.511490] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3718.802608] Lustre: server umount lustre-MDT0000 complete [ 3728.823061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3729.101564] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3729.113886] Lustre: 122089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3729.125669] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3733.742646] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3738.873688] Lustre: server umount lustre-MDT0000 complete [ 3749.163252] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 21:58:02 (1787363882) [ 3751.193176] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3753.438472] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 21:58:06 (1787363886) [ 3774.861329] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3787.159217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3793.412298] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3804.091319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3804.284180] Lustre: 124142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3804.335381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3804.341163] Lustre: Skipped 1 previous similar message [ 3804.429188] Lustre: lustre-MDT0001: new disk, initializing [ 3804.609438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3804.620299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3809.655526] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3815.640604] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3824.279509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3824.695590] Lustre: lustre-OST0000: new disk, initializing [ 3824.702545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3824.712358] Lustre: 125721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3825.959932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3825.971082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3826.135528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3832.628724] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3845.347621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3845.527382] Lustre: lustre-OST0001: new disk, initializing [ 3845.532083] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3845.539225] Lustre: 126591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3847.400170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3847.408822] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3847.526800] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3851.626331] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3862.848356] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3867.365780] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4287.146511] Lustre: 133027:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4287.154460] Lustre: 133027:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4287.172327] Lustre: 133027:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4287.187907] Lustre: 133027:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4287.194209] Lustre: 133027:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4287.203428] Lustre: 133027:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4606.340446] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:12:19 (1787364739) [ 4623.846817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4623.856456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4623.867296] Lustre: Skipped 2 previous similar messages [ 4623.871833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4623.886252] Lustre: Skipped 2 previous similar messages [ 4625.916306] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4628.867214] Lustre: server umount lustre-MDT0000 complete [ 4631.027534] LustreError: 124149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4631.070245] LustreError: 124149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 4633.179721] LustreError: 124880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787364769 with bad export cookie 15521887712232813262 [ 4633.181240] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4633.187989] LustreError: 124880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4636.133630] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.138829] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4636.159483] Lustre: Skipped 3 previous similar messages [ 4636.177475] Lustre: Skipped 5 previous similar messages [ 4639.754950] Lustre: server umount lustre-MDT0001 complete [ 4650.223128] Lustre: server umount lustre-OST0000 complete [ 4659.996074] Lustre: server umount lustre-OST0001 complete [ 4666.906262] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 4676.764440] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4721.504555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.941615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4722.001580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4722.201485] Lustre: lustre-MDT0000: new disk, initializing [ 4722.300124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4722.312270] Lustre: Skipped 5 previous similar messages [ 4722.349616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4727.496483] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4739.732162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4739.917269] Lustre: 136085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4740.052662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4740.058358] Lustre: Skipped 1 previous similar message [ 4740.120183] Lustre: lustre-MDT0001: new disk, initializing [ 4740.249916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4740.263918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4745.847693] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4756.877681] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4763.229668] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4770.894625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4771.249950] Lustre: lustre-OST0000: new disk, initializing [ 4771.253636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4771.267128] Lustre: 138053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4771.371092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4771.383851] Lustre: Skipped 1 previous similar message [ 4772.710116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4772.749166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4772.914779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4779.459845] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4792.805794] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4803.502074] Lustre: lustre-MDD0000: changelog on [ 4806.833179] Lustre: lustre-MDD0001: changelog on [ 6682.642895] Lustre: lustre-MDD0001: changelog off [ 6684.703031] Lustre: 137178:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6685.380153] Lustre: lustre-MDD0000: changelog off [ 6692.839255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6692.839814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6692.844361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6692.844370] Lustre: Skipped 2 previous similar messages [ 6692.851317] Lustre: Skipped 1 previous similar message [ 6697.957308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6697.972177] Lustre: Skipped 2 previous similar messages [ 6703.073192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6703.079794] Lustre: Skipped 1 previous similar message [ 6705.119117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6705.364808] Lustre: server umount lustre-OST0000 complete [ 6711.266307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6711.274420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6711.296599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6713.316289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6715.509492] Lustre: server umount lustre-MDT0000 complete [ 6718.443878] LustreError: 136094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6718.466228] LustreError: 136094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6718.809870] LustreError: 136080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787366854 with bad export cookie 15521887712233634537 [ 6718.814294] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6718.824470] LustreError: 136080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6719.185804] Lustre: server umount lustre-MDT0001 complete [ 6731.758471] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6734.495833] Key type lgssc unregistered [ 6734.849191] LNet: 147183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6734.860465] LNetError: 147183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6734.876088] LNet: Removed LNI 192.168.202.132@tcp [ 6735.912153] Key type .llcrypt unregistered [ 6735.914313] Key type ._llcrypt unregistered [ 6761.561774] Key type ._llcrypt registered [ 6761.563817] Key type .llcrypt registered [ 6761.712561] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 6781.021219] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6782.251961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6782.351616] alg: No test for adler32 (adler32-zlib) [ 6783.391710] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 6783.568592] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6785.200508] Key type lgssc registered [ 6786.300644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6819.942232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6819.978300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6821.258671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6821.289164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6821.375274] Lustre: lustre-MDT0000: new disk, initializing [ 6821.441927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6821.464659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6825.751457] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6836.860084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.979475] Lustre: 150391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6837.027622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6837.031624] Lustre: Skipped 1 previous similar message [ 6837.100888] Lustre: lustre-MDT0001: new disk, initializing [ 6837.292341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6837.369536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6837.377863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6842.845502] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6853.473402] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6860.019805] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6867.395570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6867.673687] Lustre: lustre-OST0000: new disk, initializing [ 6867.676946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6867.685369] Lustre: 152361:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6867.781641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6873.690537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6873.707826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6873.772722] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6873.804362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6882.849615] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6888.050721] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_module obdecho/obdecho [ 6889.825255] LustreError: 153365:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6889.830704] LustreError: 153365:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6891.461612] LustreError: 153461:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6903.777343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6903.784469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.816367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6904.806206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6904.809897] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.821342] Lustre: Skipped 1 previous similar message [ 6904.841584] Lustre: Skipped 1 previous similar message [ 6908.942505] Lustre: server umount lustre-MDT0000 complete [ 6912.581214] LustreError: 150384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367048 with bad export cookie 852298146552735425 [ 6912.594067] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6912.879591] Lustre: server umount lustre-MDT0001 complete [ 6926.881666] Lustre: server umount lustre-OST0000 complete [ 6952.877473] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 6967.195679] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7008.550771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7008.919622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7008.959943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7009.094661] Lustre: lustre-MDT0000: new disk, initializing [ 7009.188583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7009.203471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7014.614145] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7026.565704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.679502] Lustre: 157198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7026.724859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7026.727580] Lustre: Skipped 1 previous similar message [ 7026.790594] Lustre: lustre-MDT0001: new disk, initializing [ 7026.869134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7026.889132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7026.896379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7032.786222] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7043.555905] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7051.034261] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7059.294882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7059.614427] Lustre: lustre-OST0000: new disk, initializing [ 7059.620115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7059.626086] Lustre: 159169:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7059.711988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7066.170348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7066.188316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7066.317254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7067.757801] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7080.047607] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7085.168708] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7085.404943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7089.937181] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7090.195729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7097.326537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7097.329694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.338310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7097.352852] Lustre: Skipped 1 previous similar message [ 7097.390090] Lustre: Skipped 2 previous similar messages [ 7098.691822] Lustre: server umount lustre-OST0000 complete [ 7103.971825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7103.991180] LustreError: Skipped 1 previous similar message [ 7103.996199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7104.012741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7109.756102] Lustre: server umount lustre-MDT0000 complete [ 7112.672211] LustreError: 157211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7112.700777] LustreError: 157211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7113.919170] LustreError: 157191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367249 with bad export cookie 852298146552736384 [ 7113.921601] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7113.932706] LustreError: 157191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7114.318564] Lustre: server umount lustre-MDT0001 complete [ 7119.273274] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 22:54:11 (1787367251) [ 7121.695898] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7124.328911] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 22:54:16 (1787367256) [ 7132.910809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7133.508966] LustreError: 161584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7133.642357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7138.820806] LustreError: 161585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.767671] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7142.950643] LustreError: 161584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7148.006915] LustreError: 161585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7149.779903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7154.665760] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7163.399827] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7170.451391] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7178.006298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7178.158460] LustreError: 163403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7178.313606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7178.322693] Lustre: Skipped 1 previous similar message [ 7185.501055] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7194.719457] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7205.857029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7205.869118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7205.888789] Lustre: Skipped 3 previous similar messages [ 7205.897255] Lustre: Skipped 3 previous similar messages [ 7208.313806] Lustre: server umount lustre-MDT0000 complete [ 7210.978326] LustreError: 161584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7210.992044] LustreError: 161584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7212.264960] LustreError: 161566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367348 with bad export cookie 852298146552737266 [ 7212.270626] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7212.277026] LustreError: 161566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7216.097568] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7216.115965] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7216.124557] Lustre: Skipped 2 previous similar messages [ 7218.910071] Lustre: server umount lustre-MDT0001 complete [ 7226.676054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7241.963533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7242.317068] LustreError: 165426:0:(llog.c:1655:llog_backup()) MGC192.168.202.132@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7242.330326] Lustre: 165426:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.132@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7247.020196] LustreError: 165433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.057973] LustreError: 165433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7282.806941] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (not set up) [ 7282.810213] Lustre: Skipped 2 previous similar messages [ 7303.135337] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7303.139433] Lustre: 165426:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.132@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7312.487833] LustreError: 165431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.499599] LustreError: 165431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 47 previous similar messages [ 7344.111167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7349.186533] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7349.231791] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7359.258510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7372.776544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7387.657351] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7387.669611] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7396.639944] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7398.725505] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7405.190562] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7406.850892] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7428.577300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7428.582091] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7428.583224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7428.598692] Lustre: Skipped 2 previous similar messages [ 7429.783874] Lustre: server umount lustre-MDT0000 complete [ 7433.947040] LustreError: 161565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367570 with bad export cookie 852298146552743755 [ 7433.958916] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7433.972779] LustreError: 161565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7434.368849] Lustre: server umount lustre-MDT0001 complete [ 7449.497609] Lustre: server umount lustre-OST0000 complete [ 7473.891737] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 7486.269927] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7529.292977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7529.555237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7529.593835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7529.717285] Lustre: lustre-MDT0000: new disk, initializing [ 7529.859894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.870228] Lustre: Skipped 1 previous similar message [ 7529.893764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7535.557047] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7548.052318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7548.246891] Lustre: 170548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7548.600703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7554.788717] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7568.402570] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7575.666440] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7586.826759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7587.010899] Lustre: lustre-OST0000: new disk, initializing [ 7587.014452] Lustre: Skipped 1 previous similar message [ 7587.018638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7587.021753] Lustre: Skipped 2 previous similar messages [ 7587.028928] Lustre: 172516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7593.506593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7593.517520] Lustre: Skipped 1 previous similar message [ 7593.523193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7593.601356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7593.756447] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7603.577684] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7609.306516] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7609.562757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7614.244994] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7614.593179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7619.041359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7619.047143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.073346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7619.080114] Lustre: Skipped 2 previous similar messages [ 7631.327295] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7631.541972] Lustre: server umount lustre-OST0000 complete [ 7635.941652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7635.951893] LustreError: Skipped 1 previous similar message [ 7641.668880] Lustre: server umount lustre-MDT0000 complete [ 7645.156551] LustreError: 170559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7645.185760] LustreError: 170559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 58 previous similar messages [ 7646.265493] LustreError: 170539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367782 with bad export cookie 852298146552760548 [ 7646.293453] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7646.874785] Lustre: server umount lustre-MDT0001 complete [ 7656.912027] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:03:09 (1787367789) [ 7664.350233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7669.872978] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7681.052691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.502316] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7695.760616] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7702.250361] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7710.597377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7717.806908] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7726.426853] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7751.329984] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:04:44 (1787367884) [ 7757.281972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7757.306426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7757.341147] Lustre: Skipped 3 previous similar messages [ 7757.358858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7757.368467] Lustre: Skipped 8 previous similar messages [ 7762.772494] Lustre: server umount lustre-OST0000 complete [ 7781.343211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7781.566036] Lustre: server umount lustre-MDT0000 complete [ 7785.341567] LustreError: 177932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367921 with bad export cookie 852298146552761423 [ 7785.358992] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7785.365046] LustreError: 177932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7785.727480] Lustre: server umount lustre-MDT0001 complete [ 7797.385035] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 7800.629921] Key type lgssc unregistered [ 7800.897972] LNet: 178687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7800.902831] LNetError: 178687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7801.967219] LNet: Removed LNI 192.168.202.132@tcp [ 7802.967118] Key type .llcrypt unregistered [ 7802.970828] Key type ._llcrypt unregistered [ 7825.356670] Key type ._llcrypt registered [ 7825.358497] Key type .llcrypt registered [ 7825.480735] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7826.703891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7826.778123] alg: No test for adler32 (adler32-zlib) [ 7827.786719] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 7828.089257] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 7829.736151] Key type lgssc registered [ 7830.882817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7838.623463] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7838.637447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7854.239673] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7859.360125] LustreError: 179930:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 7864.479560] LustreError: 179930:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7864.494280] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7864.500031] LustreError: MGC192.168.202.132@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7864.521043] LustreError: 179930:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7864.526868] LustreError: 179930:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7864.539945] LustreError: 179930:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7864.604906] Lustre: server umount lustre-OST0000 complete [ 7864.609037] LustreError: 179930:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7874.092504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7875.677578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7880.367697] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7885.794170] LustreError: 180430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7885.814911] LustreError: 180430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7889.047921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7889.625504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7894.745481] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7902.206373] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7908.465652] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7916.491393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7916.644162] LustreError: 182246:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7916.653505] Lustre: 182246:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.132@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7916.804942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7923.194363] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7931.041153] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7934.243066] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7934.256463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7934.272546] Lustre: Skipped 1 previous similar message [ 7934.287750] Lustre: Skipped 2 previous similar messages [ 7939.554338] LustreError: 180430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7939.585211] LustreError: 180430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7939.920626] Lustre: server umount lustre-MDT0000 complete [ 7944.233509] LustreError: 180409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368080 with bad export cookie 12730833457753084267 [ 7944.241730] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7944.244479] LustreError: 180409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7944.504318] LustreError: 181159:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e8d08d86 x1874191150248192/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7944.618527] Lustre: server umount lustre-MDT0001 complete [ 7958.867508] Lustre: server umount lustre-OST0000 complete [ 7970.507500] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 7986.058772] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8029.762422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8030.031693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8030.052732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8030.140938] Lustre: lustre-MDT0000: new disk, initializing [ 8030.206572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8030.220354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8033.899227] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8046.159466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8046.370745] Lustre: 185791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8046.462639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8046.475770] Lustre: Skipped 1 previous similar message [ 8046.587536] Lustre: lustre-MDT0001: new disk, initializing [ 8046.752234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8046.803478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8046.823943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8051.796969] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8063.342534] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8069.756274] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8077.028566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8077.345650] Lustre: lustre-OST0000: new disk, initializing [ 8077.348841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8077.353849] Lustre: 187762:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8077.470120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8084.620207] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8086.683334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8086.707094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8086.819514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8096.274676] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8102.670825] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8103.036538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8108.041576] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8108.224162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8112.609598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8112.620560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8112.637976] Lustre: Skipped 1 previous similar message [ 8112.655751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8117.735812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8117.740984] Lustre: Skipped 2 previous similar messages [ 8122.852568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8122.862969] Lustre: Skipped 1 previous similar message [ 8124.383434] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8124.724283] Lustre: server umount lustre-OST0000 complete [ 8131.042510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.062397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8131.070486] Lustre: Skipped 2 previous similar messages [ 8131.084853] Lustre: Skipped 1 previous similar message [ 8138.211047] LustreError: 185803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8138.247195] LustreError: 185803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8138.595832] Lustre: server umount lustre-MDT0000 complete [ 8142.706740] LustreError: 185783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368278 with bad export cookie 12730833457753084981 [ 8142.708714] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8142.724189] LustreError: 185783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8143.061343] Lustre: server umount lustre-MDT0001 complete [ 8154.014195] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:11:27 (1787368287) [ 8176.643151] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8179.533903] Key type lgssc unregistered [ 8179.807218] LNet: 190713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8179.814439] LNetError: 190713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8179.833892] LNet: Removed LNI 192.168.202.132@tcp [ 8180.762265] Key type .llcrypt unregistered [ 8180.764181] Key type ._llcrypt unregistered [ 8204.039575] Key type ._llcrypt registered [ 8204.044094] Key type .llcrypt registered [ 8204.256169] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8205.787543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8205.947695] alg: No test for adler32 (adler32-zlib) [ 8207.151976] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 8207.368256] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8209.026076] Key type lgssc registered [ 8210.210731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8218.711087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8218.726399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8220.084918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.341942] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8230.374440] LustreError: 191932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8230.413483] LustreError: 191932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8235.491770] LustreError: 191933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8236.455693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8236.938563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8243.513578] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8255.307855] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8263.460627] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8272.775589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8273.100991] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8273.103153] LustreError: 193800:0:(llog.c:1696:llog_backup()) MGC192.168.202.132@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8273.116286] Lustre: 193800:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.132@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8273.361429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8279.966412] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8287.560114] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8291.815207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8291.828434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8291.837334] Lustre: Skipped 2 previous similar messages [ 8291.859078] Lustre: Skipped 2 previous similar messages [ 8296.133572] Lustre: server umount lustre-MDT0000 complete [ 8296.934122] LustreError: 191933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.962626] LustreError: 191933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8299.426655] LustreError: 192669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368435 with bad export cookie 11783913142279183976 [ 8299.427490] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8299.438509] LustreError: 192669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8299.777704] Lustre: server umount lustre-MDT0001 complete [ 8314.356038] Lustre: server umount lustre-OST0000 complete [ 8326.634468] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 8339.015428] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8386.311262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8386.506457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8386.528977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8386.606555] Lustre: lustre-MDT0000: new disk, initializing [ 8386.721500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8386.734825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8391.821657] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8403.614439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8403.768134] Lustre: 197330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8403.814115] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8403.816756] Lustre: Skipped 1 previous similar message [ 8403.819317] LustreError: 198112:0:(llog.c:1696:llog_backup()) MGC192.168.202.132@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8403.823991] LustreError: 198112:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8403.831023] Lustre: 198112:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.132@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8403.856832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8403.859718] Lustre: Skipped 1 previous similar message [ 8403.945301] Lustre: lustre-MDT0001: new disk, initializing [ 8404.090917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8404.146110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8404.167296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8409.239506] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8420.032768] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8426.491217] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8433.763904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8434.070586] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8434.073084] Lustre: Skipped 1 previous similar message [ 8434.076082] LustreError: 199274:0:(llog.c:1696:llog_backup()) MGC192.168.202.132@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8434.083730] LustreError: 199274:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8434.088749] Lustre: 199274:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.132@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8434.095023] Lustre: 199274:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8434.109700] Lustre: lustre-OST0000: new disk, initializing [ 8434.114948] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8434.124849] Lustre: 199297:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8434.249410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8440.410587] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8443.450836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8443.471580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8443.637962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8450.559341] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8456.006757] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8456.209989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8460.888795] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8461.140108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8464.354807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8464.358039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.365229] LustreError: Skipped 1 previous similar message [ 8464.367909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8464.394136] Lustre: Skipped 1 previous similar message [ 8469.563507] Lustre: server umount lustre-OST0000 complete [ 8474.597466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8474.605565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8474.614992] Lustre: Skipped 1 previous similar message [ 8474.621474] Lustre: Skipped 1 previous similar message [ 8479.713697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8479.726346] Lustre: Skipped 1 previous similar message [ 8479.838474] Lustre: server umount lustre-MDT0000 complete [ 8483.900474] LustreError: 197320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368619 with bad export cookie 11783913142279184690 [ 8483.901368] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8483.907809] LustreError: 197320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8484.163937] Lustre: server umount lustre-MDT0001 complete [ 8495.111988] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:17:07 (1787368627) [ 8514.888205] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8527.497886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.976338] LustreError: 202424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8528.020913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8533.431478] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8533.472544] LustreError: 202425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8533.508478] LustreError: 202425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8543.469535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8548.893966] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8552.406813] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8560.130461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8560.503650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8560.509862] Lustre: Skipped 1 previous similar message [ 8566.914384] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8576.658744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8576.788960] Lustre: lustre-OST0001: new disk, initializing [ 8576.792834] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8576.798596] Lustre: 204731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8582.802151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8582.827482] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8582.966863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8583.236869] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8594.792659] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8601.669974] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8604.102466] Lustre: DEBUG MARKER: ADD OST3 [ 8609.641231] loop: module loaded [ 8609.714023] loop0: detected capacity change from 0 to 204800000 [ 8609.982944] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8609.997966] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8610.029445] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8616.596170] loop0: detected capacity change from 0 to 204800000 [ 8617.642748] Lustre: DEBUG MARKER: STOP OST3 [ 8623.172087] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8623.400918] Lustre: lustre-OST0003: new disk, initializing [ 8623.408960] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8623.414983] Lustre: 206724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8624.559842] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8624.716917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8624.722255] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8624.761468] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8626.596799] LustreError: 206832:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8627.804340] Lustre: Failing over lustre-OST0003 [ 8627.956450] Lustre: server umount lustre-OST0003 complete [ 8629.746416] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8629.758100] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8629.784389] LustreError: 203911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8629.800334] LustreError: 203911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8631.361446] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8641.187644] Lustre: DEBUG MARKER: START OST3 again [ 8648.676828] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8649.120199] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8649.125961] Lustre: Skipped 2 previous similar messages [ 8649.141551] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8650.472907] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8650.945852] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8650.946554] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8656.850439] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8678.028319] Lustre: Failing over lustre-OST0003 [ 8678.240485] Lustre: server umount lustre-OST0003 complete [ 8679.940610] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8679.946850] LustreError: 203911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.954489] Lustre: Skipped 2 previous similar messages [ 8679.983933] LustreError: 203911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8697.313367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8697.323841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8697.357635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8703.616372] Lustre: server umount lustre-MDT0000 complete [ 8709.459119] LustreError: 205500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368845 with bad export cookie 11783913142279185572 [ 8709.488337] LustreError: 205500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8709.505196] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8710.637653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8710.660184] Lustre: Skipped 5 previous similar messages [ 8714.656484] LustreError: 203802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.669389] LustreError: 203802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8716.073108] Lustre: server umount lustre-MDT0001 complete [ 8720.433604] Lustre: server umount lustre-OST0000 complete [ 8723.974260] Lustre: server umount lustre-OST0001 complete [ 8730.925400] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 8742.883928] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8783.275331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8783.615580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8783.649544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8783.766744] Lustre: lustre-MDT0000: new disk, initializing [ 8783.872499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8783.901897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8788.517245] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8800.045663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8800.236711] Lustre: 212194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8800.346194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8800.357546] Lustre: Skipped 1 previous similar message [ 8800.466348] Lustre: lustre-MDT0001: new disk, initializing [ 8800.594569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8800.604326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8804.606656] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8814.309199] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8821.489828] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8830.155557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8830.324575] Lustre: 214162:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8831.936119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8832.194199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8837.704394] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8848.359263] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8854.025399] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8854.253324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8859.401900] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8859.606716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8862.693666] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8862.694715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8862.698497] LustreError: Skipped 1 previous similar message [ 8862.699746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8862.699751] Lustre: Skipped 2 previous similar messages [ 8862.738125] Lustre: Skipped 6 previous similar messages [ 8867.811358] LustreError: 214147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8867.934887] Lustre: server umount lustre-OST0000 complete [ 8872.420622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8878.750076] Lustre: server umount lustre-MDT0000 complete [ 8883.070680] LustreError: 212185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369019 with bad export cookie 11783913142279187861 [ 8883.082674] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8883.440769] Lustre: server umount lustre-MDT0001 complete [ 8887.708380] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:23:40 (1787369020) [ 8905.430190] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 8916.840068] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8959.110946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8959.323623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8959.342143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8959.345888] Lustre: Skipped 1 previous similar message [ 8959.442208] Lustre: lustre-MDT0000: new disk, initializing [ 8959.450646] Lustre: Skipped 1 previous similar message [ 8959.561917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8959.568190] Lustre: Skipped 1 previous similar message [ 8964.652555] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8976.051305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8976.186830] Lustre: 218771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8976.472638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8981.613234] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 8992.512866] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8999.416851] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9007.235692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9007.570910] Lustre: 220744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9008.960157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9009.149080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9013.525494] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9024.273879] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9029.815558] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9030.024684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9034.302226] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9034.521140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9039.843276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9039.850912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9039.867798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9039.876417] Lustre: Skipped 3 previous similar messages [ 9039.885928] Lustre: Skipped 6 previous similar messages [ 9042.755148] Lustre: server umount lustre-OST0000 complete [ 9061.343350] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9061.706325] Lustre: server umount lustre-MDT0000 complete [ 9065.269613] LustreError: 220743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369201 with bad export cookie 11783913142279188750 [ 9065.290460] LustreError: 220743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9065.292294] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9065.540369] Lustre: server umount lustre-MDT0001 complete [ 9073.383277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9073.946766] LustreError: 222961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9073.960896] LustreError: 222961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9074.012676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9074.018226] Lustre: Skipped 5 previous similar messages [ 9078.914732] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9087.055307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9091.086376] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9099.419110] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9106.401316] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9114.188705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9121.551944] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9129.924863] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9142.240615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9142.252397] LustreError: Skipped 2 previous similar messages [ 9142.259256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9142.259278] Lustre: Skipped 2 previous similar messages [ 9142.267495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9142.292401] Lustre: Skipped 8 previous similar messages [ 9146.576661] Lustre: server umount lustre-OST0000 complete [ 9165.281258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9165.585957] Lustre: server umount lustre-MDT0000 complete [ 9170.044750] LustreError: 224803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369306 with bad export cookie 11783913142279189632 [ 9170.048395] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9170.062032] LustreError: 224803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9170.444589] Lustre: server umount lustre-MDT0001 complete [ 9268.017309] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 9270.599497] Key type lgssc unregistered [ 9270.965198] LNet: 226628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9270.971352] LNetError: 226628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9271.002268] LNet: Removed LNI 192.168.202.132@tcp [ 9271.655893] Key type .llcrypt unregistered [ 9271.658327] Key type ._llcrypt unregistered [ 9303.277821] Key type ._llcrypt registered [ 9303.280256] Key type .llcrypt registered [ 9303.361195] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 9319.608485] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9320.912723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9320.948816] alg: No test for adler32 (adler32-zlib) [ 9321.979775] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 9322.220697] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 9323.879415] Key type lgssc registered [ 9324.875685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9364.174197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9364.193246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.567155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9365.619790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9365.747898] Lustre: lustre-MDT0000: new disk, initializing [ 9365.849091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.876962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9370.057135] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9381.610392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9381.779950] Lustre: 229998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9381.830283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9381.832457] Lustre: Skipped 1 previous similar message [ 9381.888627] Lustre: lustre-MDT0001: new disk, initializing [ 9382.001475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9382.066831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9382.086071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9387.164672] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9396.216804] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9403.404887] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9411.903196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9412.230705] Lustre: lustre-OST0000: new disk, initializing [ 9412.234282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9412.245698] Lustre: 231968:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9412.352057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9417.797100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9417.810288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9417.948859] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9419.299145] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9429.026162] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9434.370702] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9434.610593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9439.397597] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9439.639521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9443.810346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9443.811337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9443.818895] LustreError: Skipped 1 previous similar message [ 9443.820069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9443.845730] Lustre: Skipped 1 previous similar message [ 9447.925760] Lustre: server umount lustre-OST0000 complete [ 9453.537024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9453.544355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9453.553946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9453.562746] Lustre: Skipped 1 previous similar message [ 9459.178073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9459.182836] Lustre: Skipped 3 previous similar messages [ 9464.297114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9464.308142] Lustre: Skipped 1 previous similar message [ 9466.339197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9466.839331] Lustre: server umount lustre-MDT0000 complete [ 9470.796786] LustreError: 229990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369606 with bad export cookie 14816750483699865810 [ 9470.801089] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9470.804640] LustreError: 229990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9471.486395] Lustre: server umount lustre-MDT0001 complete [ 9493.648138] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9504.640514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9505.118095] LustreError: 234904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9505.139349] LustreError: 234904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9505.211610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9509.534518] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9510.369350] LustreError: 234905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9515.492278] LustreError: 234904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9517.926221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9518.297703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9522.691430] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9525.803630] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9534.208735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9534.599470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9541.686976] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9550.705591] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9550.878519] Lustre: lustre-OST0001: new disk, initializing [ 9550.888559] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9550.896785] Lustre: 237211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9551.025707] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9556.887762] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9560.749681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9560.772925] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9560.842218] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9569.165838] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9579.328952] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9593.342933] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 23:35:26 (1787369726) [ 9596.904941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9596.917625] Lustre: Skipped 3 previous similar messages [ 9596.921848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9596.931624] Lustre: Skipped 1 previous similar message [ 9601.359686] Lustre: server umount lustre-MDT0000 complete [ 9602.030849] LustreError: 234905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9602.059951] LustreError: 234905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9605.016455] LustreError: 236040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369741 with bad export cookie 14816750483699866692 [ 9605.023902] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9605.035252] LustreError: 236040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9605.505073] Lustre: server umount lustre-MDT0001 complete [ 9619.228405] Lustre: server umount lustre-OST0000 complete [ 9632.398145] Lustre: server umount lustre-OST0001 complete [ 9640.464117] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 9654.288298] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 9693.535220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9693.751866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9693.773719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9693.836400] Lustre: lustre-MDT0000: new disk, initializing [ 9693.936279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9693.972644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9698.857560] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9711.092838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9711.201506] Lustre: 241175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9711.240733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9711.243868] Lustre: Skipped 1 previous similar message [ 9711.294604] Lustre: lustre-MDT0001: new disk, initializing [ 9711.389569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9711.398763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9715.760879] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9724.970161] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9731.545970] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9739.450593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9739.717920] Lustre: lustre-OST0000: new disk, initializing [ 9739.721830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9739.728842] Lustre: 243141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9741.629208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9741.635961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9741.755477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9746.565596] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9757.322754] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9761.977732] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9762.216719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9767.861942] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9768.107333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9772.514341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9772.515410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9772.531036] Lustre: Skipped 1 previous similar message [ 9772.534574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9772.540667] LustreError: Skipped 1 previous similar message [ 9772.562328] Lustre: Skipped 3 previous similar messages [ 9776.447751] Lustre: server umount lustre-OST0000 complete [ 9782.753430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9782.765959] Lustre: Skipped 2 previous similar messages [ 9786.841661] Lustre: server umount lustre-MDT0000 complete [ 9787.876752] LustreError: 241186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9787.900700] LustreError: 241186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9790.787790] LustreError: 243153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369926 with bad export cookie 14816750483699868827 [ 9790.797893] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9790.800860] LustreError: 243153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9791.123809] Lustre: server umount lustre-MDT0001 complete [ 9801.327941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9801.620547] LustreError: 245360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9801.681637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9801.685971] Lustre: Skipped 2 previous similar messages [ 9807.381617] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9818.494685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.314431] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9834.074449] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9841.860589] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9849.213590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9857.228417] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 9865.404581] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10015.712175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10015.718882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10015.734143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10015.741499] Lustre: Skipped 1 previous similar message [10015.755666] Lustre: Skipped 3 previous similar messages [10019.511891] Lustre: server umount lustre-OST0000 complete [10038.239219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10038.551471] Lustre: server umount lustre-MDT0000 complete [10040.290462] LustreError: 245360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10040.310973] LustreError: 245360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10042.750404] LustreError: 245340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370178 with bad export cookie 14816750483699869709 [10042.763268] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10043.247893] Lustre: server umount lustre-MDT0001 complete [10134.730503] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10137.479125] Key type lgssc unregistered [10137.792783] LNet: 248977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10137.799671] LNetError: 248977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10137.812993] LNet: Removed LNI 192.168.202.132@tcp [10138.651160] Key type .llcrypt unregistered [10138.653432] Key type ._llcrypt unregistered [10165.699921] Key type ._llcrypt registered [10165.702457] Key type .llcrypt registered [10165.816170] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [10183.239763] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10184.685729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10184.773365] alg: No test for adler32 (adler32-zlib) [10185.841963] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [10186.087244] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [10187.799242] Key type lgssc registered [10189.078348] Lustre: Echo OBD driver; http://www.lustre.org/ [10238.716573] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10249.856129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10249.885423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10251.296242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10251.324807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10251.427516] Lustre: lustre-MDT0000: new disk, initializing [10251.575673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10251.596284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10256.290455] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10265.700847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10265.851070] Lustre: 253051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10265.922560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10265.933267] Lustre: Skipped 1 previous similar message [10266.019670] Lustre: lustre-MDT0001: new disk, initializing [10266.129558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10266.170262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10266.185161] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10270.908147] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10276.094923] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10283.273422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10283.547424] Lustre: lustre-OST0000: new disk, initializing [10283.557916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10283.565084] Lustre: 254681:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10283.707249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10288.715075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10288.731772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10288.896085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10290.072391] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10300.890966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10301.138308] Lustre: lustre-OST0001: new disk, initializing [10301.140840] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10301.145746] Lustre: 255554:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10301.237239] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10307.660856] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10309.730062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10309.739931] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10309.837918] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10319.481868] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10327.446963] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10340.835843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10340.840611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10340.855462] Lustre: Skipped 2 previous similar messages [10340.864720] Lustre: Skipped 3 previous similar messages [10345.965097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10345.972135] Lustre: Skipped 3 previous similar messages [10351.082770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10351.102848] Lustre: Skipped 3 previous similar messages [10352.095226] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10352.496178] Lustre: server umount lustre-MDT0000 complete [10355.793137] LustreError: 253041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370491 with bad export cookie 8599821010040066833 [10355.810216] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10356.121661] Lustre: server umount lustre-MDT0001 complete [10369.093617] Lustre: server umount lustre-OST0000 complete [10372.577769] Lustre: 250554:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787370492/real 1787370492] req@ffffa0fef7fd9180 x1874193622838656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787370508 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10372.631475] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10372.640350] Lustre: Skipped 1 previous similar message [10373.357655] Lustre: server umount lustre-OST0001 complete [10389.351189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10398.007098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10407.194045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10416.945081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10435.693769] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10445.012506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10445.026897] Lustre: lustre-MDT0000: reset Object Index mappings [10445.410930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10449.054298] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10450.917949] LustreError: 260565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10450.941174] LustreError: 260565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10455.010421] LustreError: 260566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10456.108337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10456.130304] Lustre: lustre-MDT0001: reset Object Index mappings [10456.526556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10461.431214] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10464.819121] Lustre: 261708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10472.546928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10473.101383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10479.862951] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10481.323755] LustreError: 262061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10486.764522] LustreError: 262481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10490.865562] LustreError: 262062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10490.889430] LustreError: 262062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10490.890678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10491.725147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10497.552524] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10499.306876] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10507.018800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10510.536984] Lustre: 263580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10517.406186] Lustre: server umount lustre-MDT0000 complete [10518.001766] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10518.011973] LustreError: 260566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10518.018246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10518.035028] Lustre: Skipped 2 previous similar messages [10518.071905] LustreError: 260566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10520.850014] LustreError: 261308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370656 with bad export cookie 8599821010040073623 [10520.863510] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10520.865528] LustreError: 261308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10521.323138] Lustre: server umount lustre-MDT0001 complete [10535.371213] Lustre: server umount lustre-OST0000 complete [10548.762594] Lustre: server umount lustre-OST0001 complete [10565.672206] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 23:51:38 (1787370698) [10584.358232] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [10598.329465] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10636.295242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10636.595761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10636.634745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10636.716600] Lustre: lustre-MDT0000: new disk, initializing [10636.830792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10636.832923] Lustre: Skipped 1 previous similar message [10636.845925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10641.175855] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10654.709655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10654.865553] Lustre: 267854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10654.967430] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10654.971977] Lustre: Skipped 1 previous similar message [10655.112078] Lustre: lustre-MDT0001: new disk, initializing [10655.298091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10655.309680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10661.504577] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10672.209599] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10678.103229] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10685.568790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10685.842876] Lustre: lustre-OST0000: new disk, initializing [10685.847452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10685.854536] Lustre: 269822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10687.310390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10687.324474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10687.437872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10691.716580] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10700.755727] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10705.161356] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10705.362509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10709.320924] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10709.508786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10713.059994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10713.068446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10713.069104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10713.076422] Lustre: Skipped 2 previous similar messages [10718.191181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10718.199977] Lustre: Skipped 2 previous similar messages [10725.343164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10725.563063] Lustre: server umount lustre-OST0000 complete [10732.000514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10732.016616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10732.049333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10732.060675] Lustre: Skipped 2 previous similar messages [10736.727775] Lustre: server umount lustre-MDT0000 complete [10738.687080] LustreError: 268945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10738.706485] LustreError: 268945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10740.398908] LustreError: 267846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370876 with bad export cookie 8599821010040075709 [10740.414902] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10740.426456] LustreError: 267846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10740.981703] Lustre: server umount lustre-MDT0001 complete [10759.864365] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10770.828725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10771.360825] LustreError: 272756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10771.405568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10771.411527] Lustre: Skipped 2 previous similar messages [10776.423367] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10786.243746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10791.969353] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10794.961069] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10802.897735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10809.267175] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10818.091309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10818.202683] Lustre: lustre-OST0001: new disk, initializing [10818.205693] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10818.210165] Lustre: 275064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10820.284829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10820.298915] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10820.368121] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10824.795304] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10835.755754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10839.958981] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10846.211114] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10846.264049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10856.420022] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10856.427781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10856.450865] Lustre: Skipped 2 previous similar messages [10856.458765] Lustre: Skipped 2 previous similar messages [10860.646271] Lustre: server umount lustre-MDT0000 complete [10861.540541] LustreError: 272756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.579176] LustreError: 272756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10865.109543] LustreError: 272736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371001 with bad export cookie 8599821010040076598 [10865.118336] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10865.126864] LustreError: 272736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10865.897755] Lustre: server umount lustre-MDT0001 complete [10880.647850] Lustre: server umount lustre-OST0000 complete [10882.591207] Lustre: 250554:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787371002/real 1787371002] req@ffffa0feffa03b80 x1874193622998912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787371018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10882.619085] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10882.642104] Lustre: Skipped 1 previous similar message [10884.456466] Lustre: server umount lustre-OST0001 complete [10900.937552] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [10909.293176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10909.725677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10909.734598] Lustre: Skipped 3 previous similar messages [10913.729701] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10924.009491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10929.201519] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10932.253292] Lustre: 279262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10941.137045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10948.705349] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10955.798137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10958.771029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10964.480483] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10965.992278] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [10975.355739] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10980.202878] Lustre: 281138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10985.481162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10985.536279] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10995.944673] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 23:58:49 (1787371129) [11011.329778] Lustre: server umount lustre-MDT0000 complete [11011.553592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11011.559037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11011.580478] LustreError: 278122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11011.598107] LustreError: 278122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11015.358992] LustreError: 278103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371151 with bad export cookie 8599821010040083129 [11015.373454] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11015.652118] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11015.659263] Lustre: Skipped 3 previous similar messages [11017.759988] Lustre: server umount lustre-MDT0001 complete [11027.587163] Lustre: server umount lustre-OST0000 complete [11032.604846] Lustre: server umount lustre-OST0001 complete [11042.359383] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [11053.800886] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [11107.205964] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [11117.701623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11117.959820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11118.010439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11118.148585] Lustre: lustre-MDT0000: new disk, initializing [11118.252254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11122.467950] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11134.169897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11134.311525] Lustre: 285398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11134.571132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11140.129893] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11152.470344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11152.688220] Lustre: 287046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11154.271471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11158.739976] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11170.973349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11171.322509] Lustre: 287916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11171.492588] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11171.500048] Lustre: Skipped 6 previous similar messages [11172.993498] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11173.009956] Lustre: Skipped 1 previous similar message [11173.203850] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11181.160211] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11193.256792] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11197.196243] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11197.199581] Lustre: Skipped 1 previous similar message [11213.644496] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:02:27 (1787371347) [11216.355091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11216.363208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11216.379142] Lustre: Skipped 4 previous similar messages [11216.385811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11218.185321] Lustre: server umount lustre-MDT0000 complete [11221.666265] LustreError: 285391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371357 with bad export cookie 8599821010040090752 [11221.671990] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11221.675982] LustreError: 285391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11221.944482] Lustre: server umount lustre-MDT0001 complete [11236.350254] Lustre: server umount lustre-OST0000 complete [11249.175258] Lustre: server umount lustre-OST0001 complete [11256.750237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.776272] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11269.205095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11273.949736] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11281.475769] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11287.154965] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11294.085363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11294.180872] LustreError: 291993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11294.205117] LustreError: 291993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11301.211228] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11308.267165] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11323.577528] loop1: detected capacity change from 0 to 204800000 [11323.888242] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11323.903407] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11323.921867] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11329.187455] loop1: detected capacity change from 0 to 204800000 [11335.374276] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11335.467192] LustreError: 290154:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs27313: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11335.478601] LustreError: 290154:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs27313 [11335.484919] LustreError: 290154:0:(mgs_handler.c:517:mgs_target_reg()) fs27313-MDT0001: Error checking target index: rc = -13 [11335.494432] LustreError: fs27313-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11335.504763] LustreError: 293883:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11335.509709] LustreError: 293883:0:(tgt_mount.c:2129:server_put_super()) no obd fs27313-MDT0001 [11335.513411] LustreError: 293883:0:(tgt_mount.c:116:server_deregister_mount()) fs27313-MDT0001 not registered [11335.553931] Lustre: server umount fs27313-MDT0001 complete [11335.556717] LustreError: 293883:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11344.139714] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11344.210944] Lustre: Setting parameter fs27313-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs27313-MDT0001 [11344.276207] Lustre: srv-fs27313-MDT0001: No data found on store. Initialize space. [11344.281064] Lustre: Skipped 4 previous similar messages [11344.393553] Lustre: fs27313-MDT0001: new disk, initializing [11344.396684] Lustre: Skipped 3 previous similar messages [11344.552816] Lustre: srv-fs27313-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11344.560696] Lustre: 294349:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27313-MDT0001-mdtlov: can't create new llog: rc = -115 [11344.578462] LustreError: 294349:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27313-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11348.536337] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11354.441738] Lustre: server umount fs27313-MDT0001 complete [11366.488207] loop1: detected capacity change from 0 to 204800000 [11372.186519] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11372.583152] Lustre: 295916:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27313-MDT0001-mdtlov: can't create new llog: rc = -115 [11372.589768] LustreError: 295916:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27313-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11375.602403] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11377.172741] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11382.046444] Lustre: server umount fs27313-MDT0001 complete [11391.622562] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [11392.481224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11392.505043] LustreError: Skipped 1 previous similar message [11392.515133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11392.531369] Lustre: Skipped 3 previous similar messages [11397.555886] Lustre: server umount lustre-MDT0000 complete [11406.238683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11406.337905] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11411.957985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11411.976668] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11420.730447] loop1: detected capacity change from 0 to 204800000 [11426.740785] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11427.172255] Lustre: 298365:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27313-MDT0001-mdtlov: can't create new llog: rc = -115 [11427.192359] LustreError: 298365:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27313-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11432.260874] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11438.136766] Lustre: server umount fs27313-MDT0001 complete [11468.277259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11473.127227] Lustre: server umount lustre-OST0000 complete [11483.621245] Lustre: server umount lustre-MDT0000 complete [11487.834275] LustreError: 296910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371623 with bad export cookie 8599821010040093440 [11487.850728] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11487.855579] LustreError: 296910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11488.195348] Lustre: server umount lustre-MDT0001 complete [11501.799434] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [11504.526954] Key type lgssc unregistered [11504.794566] LNet: 300714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11504.803815] LNetError: 300714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11504.833691] LNet: Removed LNI 192.168.202.132@tcp [11505.720370] Key type .llcrypt unregistered [11505.723855] Key type ._llcrypt unregistered [11530.852134] Key type ._llcrypt registered [11530.859350] Key type .llcrypt registered [11531.038815] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [11532.154518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11532.265463] alg: No test for adler32 (adler32-zlib) [11533.318701] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [11533.590241] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [11535.271187] Key type lgssc registered [11536.340364] Lustre: Echo OBD driver; http://www.lustre.org/ [11544.027595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11544.063762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11545.633963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11551.130170] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11555.817702] LustreError: 301987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11555.852620] LustreError: 301987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11560.224840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11560.540753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11565.474976] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11573.912879] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11580.232257] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11585.881729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11586.196175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11590.324852] LustreError: 303815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11592.030248] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11595.752691] LustreError: 303818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11599.719720] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11600.872193] LustreError: 303815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11604.971570] LustreError: 303815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11604.998343] LustreError: 303815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11615.218623] LustreError: 304087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11615.245281] LustreError: 304087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11620.652993] loop1: detected capacity change from 0 to 204800000 [11620.960853] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11621.010386] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11621.024343] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11627.511588] loop1: detected capacity change from 0 to 204800000 [11632.241106] LustreError: 303817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11632.263301] LustreError: 303817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11633.872306] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11634.047832] Lustre: fs19608-OST0000: new disk, initializing [11634.051625] Lustre: srv-fs19608-OST0000: No data found on store. Initialize space. [11634.060977] Lustre: 305723:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11634.161643] Lustre: fs19608-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11642.806844] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11653.177266] Lustre: server umount fs19608-OST0000 complete [11666.406491] LustreError: 304087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11666.444950] LustreError: 304087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11667.559650] loop1: detected capacity change from 0 to 204800000 [11671.551973] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11673.857714] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11674.081070] Lustre: fs19608-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11680.146231] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11686.885266] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11688.197735] loop2: detected capacity change from 0 to 204800000 [11688.361663] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11688.368836] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11688.387675] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11693.178207] loop2: detected capacity change from 0 to 204800000 [11699.098837] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11699.242276] LustreError: fs19608-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11699.258333] LustreError: fs19608-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11699.267841] LustreError: 308764:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11699.270954] LustreError: 308764:0:(tgt_mount.c:2129:server_put_super()) no obd fs19608-OST0001 [11699.277493] LustreError: 308764:0:(tgt_mount.c:116:server_deregister_mount()) fs19608-OST0001 not registered [11699.333874] Lustre: server umount fs19608-OST0001 complete [11699.339455] LustreError: 308764:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11709.344237] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11709.458417] Lustre: fs19608-OST0001: new disk, initializing [11709.463457] Lustre: srv-fs19608-OST0001: No data found on store. Initialize space. [11709.468460] Lustre: 309316:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11709.551671] Lustre: fs19608-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11715.488599] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11723.974970] Lustre: server umount fs19608-OST0000 complete [11730.922441] LustreError: 304491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11730.941922] LustreError: 304491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11733.894190] Lustre: server umount fs19608-OST0001 complete [11745.340364] loop1: detected capacity change from 0 to 204800000 [11750.784808] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11750.962695] Lustre: fs19608-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11755.844938] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11761.864800] loop2: detected capacity change from 0 to 204800000 [11767.216719] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11773.149848] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11782.481554] Lustre: server umount fs19608-OST0000 complete [11790.626233] Lustre: server umount fs19608-OST0001 complete [11804.646793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11804.661093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11804.678390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11805.667557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11805.690874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11807.712111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11807.718767] Lustre: Skipped 1 previous similar message [11809.127664] Lustre: server umount lustre-OST0000 complete [11816.416989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11816.435735] LustreError: Skipped 1 previous similar message [11816.462279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.505379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11823.077611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11823.084249] Lustre: Skipped 3 previous similar messages [11828.191338] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11828.472575] Lustre: server umount lustre-MDT0000 complete [11832.234817] LustreError: 301967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371968 with bad export cookie 14895341487315687670 [11832.239208] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11832.244150] LustreError: 301967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11832.919432] Lustre: server umount lustre-MDT0001 complete [11846.609733] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [11849.503461] Key type lgssc unregistered [11849.743784] LNet: 315242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11849.747616] LNetError: 315242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11849.756962] LNet: Removed LNI 192.168.202.132@tcp [11850.707800] Key type .llcrypt unregistered [11850.709921] Key type ._llcrypt unregistered [11887.402368] Key type ._llcrypt registered [11887.405732] Key type .llcrypt registered [11887.494397] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [11905.542319] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [11907.459182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11907.491214] alg: No test for adler32 (adler32-zlib) [11908.738624] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [11909.021794] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [11910.742950] Key type lgssc registered [11912.229302] Lustre: Echo OBD driver; http://www.lustre.org/ [11952.433341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11952.472628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11953.884631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11953.980857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11954.111123] Lustre: lustre-MDT0000: new disk, initializing [11954.248207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11954.278983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11959.423273] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11970.554833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11970.694332] Lustre: 319101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11970.733122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11970.736110] Lustre: Skipped 1 previous similar message [11970.830261] Lustre: lustre-MDT0001: new disk, initializing [11970.964572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11971.026062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11971.034868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11975.616973] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [11985.658417] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11992.572142] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12000.367120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12000.605851] Lustre: lustre-OST0000: new disk, initializing [12000.607659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12000.619370] Lustre: 321073:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12000.731369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12008.594253] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12009.577442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12009.602370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12009.741045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12018.723060] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12021.028874] Lustre: Mounted lustre-client - version 2.17.57_45_g4630995 [12022.070383] Lustre: Unmounted lustre-client [12035.552485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12035.560569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12035.571043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12035.576012] Lustre: Skipped 1 previous similar message [12035.592941] Lustre: Skipped 1 previous similar message [12036.581380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12036.604730] Lustre: Skipped 1 previous similar message [12038.681988] Lustre: server umount lustre-OST0000 complete [12042.727077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12042.739761] LustreError: Skipped 1 previous similar message [12042.742177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12042.753397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12046.825146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12046.830070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12049.119888] Lustre: server umount lustre-MDT0000 complete [12053.583398] LustreError: 321072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372189 with bad export cookie 11699999434836228974 [12053.591671] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12053.601721] LustreError: 321072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12054.007379] Lustre: server umount lustre-MDT0001 complete [12067.215699] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12070.269367] Key type lgssc unregistered [12079.007546] LNet: 1 local NIs in recovery (showing 1): 192.168.202.132@tcp [12079.840511] LNet: 323236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12079.856101] LNetError: 323236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12150.503966] LNet: Removed LNI 192.168.202.132@tcp [12151.233807] Key type .llcrypt unregistered [12151.235855] Key type ._llcrypt unregistered [12169.619846] Key type ._llcrypt registered [12169.622080] Key type .llcrypt registered [12169.727098] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [12190.527893] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12191.838523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12191.985039] alg: No test for adler32 (adler32-zlib) [12193.260480] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [12193.600696] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [12195.231327] Key type lgssc registered [12196.303766] Lustre: Echo OBD driver; http://www.lustre.org/ [12228.930335] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:19:22 (1787372362) [12235.856736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12235.886969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12237.141487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12237.173305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12237.241735] Lustre: lustre-MDT0000: new disk, initializing [12237.335700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12237.354628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12241.860611] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12252.863700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12252.981295] Lustre: 326461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12253.018198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12253.021812] Lustre: Skipped 1 previous similar message [12253.078269] Lustre: lustre-MDT0001: new disk, initializing [12253.162857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12253.203622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12253.209583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12257.439326] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12265.995877] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12271.961573] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12280.361269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12280.828995] Lustre: lustre-OST0000: new disk, initializing [12280.833624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12280.842882] Lustre: 328430:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12280.963696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12288.242210] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12288.605369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12288.621820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12288.759656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12297.664938] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12314.592412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12314.597703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12314.603133] LustreError: Skipped 1 previous similar message [12314.604184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12314.632497] Lustre: Skipped 1 previous similar message [12318.494767] Lustre: server umount lustre-OST0000 complete [12324.840843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12324.847540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12324.857310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12324.870180] Lustre: Skipped 1 previous similar message [12324.892830] Lustre: Skipped 2 previous similar messages [12328.543486] Lustre: server umount lustre-MDT0000 complete [12332.259962] LustreError: 326455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372468 with bad export cookie 971288161944869327 [12332.274657] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12332.275621] LustreError: 326455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12332.741799] Lustre: server umount lustre-MDT0001 complete [12344.682911] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [12347.665024] Key type lgssc unregistered [12348.000892] LNet: 330339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12348.011937] LNetError: 330339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12348.037441] LNet: Removed LNI 192.168.202.132@tcp [12348.965765] Key type .llcrypt unregistered [12348.968399] Key type ._llcrypt unregistered [12372.601749] Key type ._llcrypt registered [12372.603721] Key type .llcrypt registered [12372.696190] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [12373.818983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12373.841232] alg: No test for adler32 (adler32-zlib) [12374.930287] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [12375.086370] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [12376.720554] Key type lgssc registered [12377.882848] Lustre: Echo OBD driver; http://www.lustre.org/ [12385.492639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12385.507972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12386.954700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12391.205158] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12396.524424] LustreError: 331615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12396.536260] LustreError: 331615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12400.415274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12400.943830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12405.442729] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12413.197308] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12420.079866] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12428.840225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12429.283267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12435.125673] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12442.330327] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12454.268844] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12455.695383] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12459.644549] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [12461.024790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12461.028545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12461.039148] Lustre: Skipped 2 previous similar messages [12461.045133] Lustre: Skipped 1 previous similar message [12464.750273] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [12465.703807] Lustre: server umount lustre-MDT0000 complete [12466.153070] LustreError: 331615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12466.160957] LustreError: 331615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12469.045652] LustreError: 331596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372605 with bad export cookie 9000578920967813775 [12469.055900] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12469.061823] LustreError: 331596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12469.866994] LustreError: 333872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.904848] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [12471.268447] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12474.287400] LustreError: 331612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12474.289735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12474.313661] LustreError: 331612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12474.323218] Lustre: Skipped 1 previous similar message [12475.537476] Lustre: server umount lustre-MDT0001 complete [12479.022962] Lustre: server umount lustre-OST0000 complete [12487.219453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12492.290785] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12495.673789] Lustre: 335795:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.132@tcp [12498.334120] Lustre: server umount MGS complete [12508.479568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12509.043980] LustreError: 336367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12509.063996] LustreError: 336367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12509.136564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12513.939107] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [12519.400762] LustreError: 336368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12519.419549] LustreError: 336368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12523.451675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc