[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488499329 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002413] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007454] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.021787] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025809] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] *** VALIDATE tmpfs *** [ 0.029413] *** VALIDATE proc *** [ 0.030198] *** VALIDATE cgroup *** [ 0.031007] *** VALIDATE cgroup2 *** [ 0.032249] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033131] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035023] Spectre V2 : User space: Vulnerable [ 0.037005] Speculative Store Bypass: Vulnerable [ 0.040227] debug: unmapping init [mem 0xffffffffa6e59000-0xffffffffa6e60fff] [ 0.042168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043605] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044018] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.046008] ... generic registers: 4 [ 0.047007] ... value mask: 0000ffffffffffff [ 0.048009] ... max period: 00007fffffffffff [ 0.049009] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.052203] rcu: Hierarchical SRCU implementation. [ 0.054530] smp: Bringing up secondary CPUs ... [ 0.055472] x86: Booting SMP configuration: [ 0.056018] .... node #0, CPUs: #1 #2 #3 [ 0.060012] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.241032] node 0 deferred pages initialised in 173ms [ 0.244194] devtmpfs: initialized [ 0.245806] x86/mm: Memory block size: 128MB [ 0.252092] gcov: version magic: 0x41383552 [ 0.255351] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.260115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.264478] pinctrl core: initialized pinctrl subsystem [ 0.265267] [ 0.267078] ************************************************************* [ 0.269019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.273019] ** ** [ 0.276023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.278020] ** ** [ 0.281018] ** This means that this kernel is built to expose internal ** [ 0.284016] ** IOMMU data structures, which may compromise security on ** [ 0.286012] ** your system. ** [ 0.288031] ** ** [ 0.291015] ** If you see this message and you are not debugging the ** [ 0.293015] ** kernel, report this immediately to your vendor! ** [ 0.296029] ** ** [ 0.298016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.301017] ************************************************************* [ 0.304597] NET: Registered protocol family 16 [ 0.307679] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.311085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.315084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.319589] cpuidle: using governor menu [ 0.320849] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.325604] PCI: Using configuration type 1 for base access [ 0.328143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.339052] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.341071] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.346044] cryptd: max_cpu_qlen set to 1000 [ 0.347258] ACPI: Added _OSI(Module Device) [ 0.348013] ACPI: Added _OSI(Processor Device) [ 0.350018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.352021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.357739] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.365712] ACPI: Interpreter enabled [ 0.368086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.370015] ACPI: Using IOAPIC for interrupt routing [ 0.373144] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.378723] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.391645] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.395064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.398021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.401100] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.407515] acpiphp: Slot [2] registered [ 0.410197] acpiphp: Slot [5] registered [ 0.412629] acpiphp: Slot [6] registered [ 0.415203] acpiphp: Slot [7] registered [ 0.417541] acpiphp: Slot [8] registered [ 0.419182] acpiphp: Slot [9] registered [ 0.421285] acpiphp: Slot [10] registered [ 0.423177] acpiphp: Slot [3] registered [ 0.425225] acpiphp: Slot [4] registered [ 0.427127] acpiphp: Slot [11] registered [ 0.429160] acpiphp: Slot [12] registered [ 0.431129] acpiphp: Slot [13] registered [ 0.433136] acpiphp: Slot [14] registered [ 0.435176] acpiphp: Slot [15] registered [ 0.437209] acpiphp: Slot [16] registered [ 0.439146] acpiphp: Slot [17] registered [ 0.441138] acpiphp: Slot [18] registered [ 0.442163] acpiphp: Slot [19] registered [ 0.444214] acpiphp: Slot [20] registered [ 0.446134] acpiphp: Slot [21] registered [ 0.448189] acpiphp: Slot [22] registered [ 0.450136] acpiphp: Slot [23] registered [ 0.451131] acpiphp: Slot [24] registered [ 0.453170] acpiphp: Slot [25] registered [ 0.455230] acpiphp: Slot [26] registered [ 0.457132] acpiphp: Slot [27] registered [ 0.458114] acpiphp: Slot [28] registered [ 0.460166] acpiphp: Slot [29] registered [ 0.461182] acpiphp: Slot [30] registered [ 0.463190] acpiphp: Slot [31] registered [ 0.465107] PCI host bridge to bus 0000:00 [ 0.467027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.469044] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.472085] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.475033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.478030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.481035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.483316] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.487000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.490433] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.501012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.507063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.510029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.513026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.518028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.521276] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.525919] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.529055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.535220] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.541020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.556022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.562893] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.568910] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.581018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.589020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.613021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.629213] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.638020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.653022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.680023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.693418] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.702022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.709020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.728027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.743719] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.758021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.792021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.816024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.834453] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.845021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.853022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.874023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.890060] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.902000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.917014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.945022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.965472] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.968563] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.971471] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.974489] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.977339] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.983085] iommu: Default domain type: Passthrough [ 0.985366] SCSI subsystem initialized [ 0.987192] ACPI: bus type USB registered [ 0.989150] usbcore: registered new interface driver usbfs [ 0.991119] usbcore: registered new interface driver hub [ 0.993107] usbcore: registered new device driver usb [ 0.995231] pps_core: LinuxPPS API ver. 1 registered [ 0.997017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.000083] PTP clock support registered [ 1.003196] EDAC MC: Ver: 3.0.0 [ 1.006178] PCI: Using ACPI for IRQ routing [ 1.009646] NetLabel: Initializing [ 1.011014] NetLabel: domain hash size = 128 [ 1.013014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.015082] NetLabel: unlabeled traffic allowed by default [ 1.018059] vgaarb: loaded [ 1.019393] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.023023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.030488] clocksource: Switched to clocksource kvm-clock [ 1.162940] VFS: Disk quotas dquot_6.6.0 [ 1.164910] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.167742] *** VALIDATE ramfs *** [ 1.169505] *** VALIDATE hugetlbfs *** [ 1.171791] pnp: PnP ACPI init [ 1.175042] pnp: PnP ACPI: found 6 devices [ 1.195344] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.199534] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.202403] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.205679] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.208198] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.211439] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.215077] NET: Registered protocol family 2 [ 1.217682] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.222668] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.226804] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.232926] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.236863] TCP: Hash tables configured (established 65536 bind 65536) [ 1.240817] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.244415] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.247580] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.251121] NET: Registered protocol family 1 [ 1.254224] RPC: Registered named UNIX socket transport module. [ 1.256805] RPC: Registered udp transport module. [ 1.259058] RPC: Registered tcp transport module. [ 1.261147] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.263835] NET: Registered protocol family 44 [ 1.265882] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.268228] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.270525] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.273123] PCI: CLS 0 bytes, default 64 [ 1.274937] Unpacking initramfs... [ 2.714495] debug: unmapping init [mem 0xffff9392bcc54000-0xffff9392bffbffff] [ 2.721322] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.723958] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.728491] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.230105] Initialise system trusted keyrings [ 3.233142] Key type blacklist registered [ 3.235422] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.247331] zbud: loaded [ 3.250738] *** VALIDATE nfs *** [ 3.252463] *** VALIDATE nfs4 *** [ 3.254415] pstore: using deflate compression [ 3.258455] Platform Keyring initialized [ 3.376972] NET: Registered protocol family 38 [ 3.379118] Key type asymmetric registered [ 3.381084] Asymmetric key parser 'x509' registered [ 3.384377] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.388094] io scheduler mq-deadline registered [ 3.390455] io scheduler kyber registered [ 3.392163] io scheduler bfq registered [ 3.395068] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.399062] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.403464] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.406962] ACPI: Power Button [PWRF] [ 3.412802] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.421106] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.435927] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.445430] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.466785] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.499121] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.529313] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.535449] Non-volatile memory driver v1.3 [ 3.536912] Linux agpgart interface v0.103 [ 3.567797] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.570701] vda: detected capacity change from 0 to 76775424 [ 3.587089] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.591290] vdb: detected capacity change from 0 to 1073741824 [ 3.608335] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.612120] vdc: detected capacity change from 0 to 2621440000 [ 3.627430] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.630709] vdd: detected capacity change from 0 to 2621440000 [ 3.654697] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.658753] vde: detected capacity change from 0 to 4294967296 [ 3.676084] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.679068] vdf: detected capacity change from 0 to 4294967296 [ 3.690174] libphy: Fixed MDIO Bus: probed [ 3.695449] usbcore: registered new interface driver usbserial_generic [ 3.697682] usbserial: USB Serial support registered for generic [ 3.700404] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.705723] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.708226] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.711657] mousedev: PS/2 mouse device common for all mice [ 3.715260] rtc_cmos 00:05: RTC can wake from S4 [ 3.718434] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.721825] rtc_cmos 00:05: registered as rtc0 [ 3.721887] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.721933] intel_pstate: CPU model not supported [ 3.728195] hid: raw HID events driver (C) Jiri Kosina [ 3.731690] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.732180] usbcore: registered new interface driver usbhid [ 3.736096] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.736731] usbhid: USB HID core driver [ 3.740206] drop_monitor: Initializing network drop monitor service [ 3.741790] Initializing XFRM netlink socket [ 3.743604] NET: Registered protocol family 10 [ 3.746989] Segment Routing with IPv6 [ 3.749575] NET: Registered protocol family 17 [ 3.751635] mpls_gso: MPLS GSO support [ 3.756933] RAS: Correctable Errors collector initialized. [ 3.759655] AVX version of gcm_enc/dec engaged. [ 3.761844] AES CTR mode by8 optimization enabled [ 3.853547] sched_clock: Marking stable (3853508159, 0)->(4788570331, -935062172) [ 3.860360] registered taskstats version 1 [ 3.862854] Loading compiled-in X.509 certificates [ 3.865559] zswap: loaded using pool lzo/zbud [ 3.896594] Key type big_key registered [ 3.909087] Key type encrypted registered [ 3.911755] ima: No TPM chip found, activating TPM-bypass! [ 3.914655] ima: Allocated hash algorithm: sha1 [ 3.917111] ima: No architecture policies found [ 3.920097] evm: Initialising EVM extended attributes: [ 3.923561] evm: security.selinux [ 3.925584] evm: security.ima [ 3.927113] evm: security.capability [ 3.929318] evm: HMAC attrs: 0x1 [ 3.933126] rtc_cmos 00:05: setting system clock to 2026-09-17 03:40:42 UTC (1789616442) [ 3.944889] debug: unmapping init [mem 0xffffffffa7e03000-0xffffffffa7ffffff] [ 3.949021] debug: unmapping init [mem 0xffffffffa6b82000-0xffffffffa6e58fff] [ 3.959095] Write protecting the kernel read-only data: 28672k [ 3.966252] debug: unmapping init [mem 0xffffffffa5203000-0xffffffffa53fffff] [ 3.969095] debug: unmapping init [mem 0xffffffffa5b14000-0xffffffffa5bfffff] [ 4.012071] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.026326] systemd[1]: Detected virtualization kvm. [ 4.030355] systemd[1]: Detected architecture x86-64. [ 4.032613] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.072433] systemd[1]: No hostname configured. [ 4.076148] systemd[1]: Set hostname to . [ 4.081460] random: systemd: uninitialized urandom read (16 bytes read) [ 4.084779] systemd[1]: Initializing machine ID from random generator. [ 4.229811] random: systemd: uninitialized urandom read (16 bytes read) [ 4.232516] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.236884] random: systemd: uninitialized urandom read (16 bytes read) [ 4.239671] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.244396] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.019757] device-mapper: uevent: version 1.0.3 [ 5.022414] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.865350] random: fast init done [ 5.867828] virtio_net virtio0 ens2: renamed from eth0 [ 5.952967] scsi host0: ata_piix [ 5.990853] scsi host1: ata_piix [ 5.995591] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.998600] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.835107] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.437932] random: crng init done [ 10.440496] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.124769] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.523341] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.845698] SELinux: Disabled at runtime. [ 12.922043] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.934715] systemd[1]: Detected virtualization kvm. [ 12.938210] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.590196] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.594978] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.600958] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.606990] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.611299] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.619266] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.624468] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ 13.759855] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 14.216569] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.560185] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.590481] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.742496] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.755257] EDAC sbridge: Ver: 1.1.2 [ 16.715547] Key type dns_resolver registered [ 17.051244] NFS: Registering the id_resolver key type [ 17.053650] Key type id_resolver registered [ 17.055380] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg645-server login: [ 61.099667] libcfs: loading out-of-tree module taints kernel. [ 61.138836] Key type ._llcrypt registered [ 61.143227] Key type .llcrypt registered [ 61.280308] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 80.486796] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 82.303353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.331588] alg: No test for adler32 (adler32-zlib) [ 83.946225] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 84.643276] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 86.464302] Key type lgssc registered [ 88.308374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.331894] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 113.745051] hrtimer: interrupt took 3180115 ns [ 146.939338] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 159.184738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.219120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.511125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.556771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 160.635145] Lustre: lustre-MDT0000: new disk, initializing [ 160.750654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.765550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.139792] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.062101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.183037] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.205233] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.209554] Lustre: Skipped 1 previous similar message [ 178.288696] Lustre: lustre-MDT0001: new disk, initializing [ 178.387498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.420739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.434542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.096725] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.907244] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.206447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.620620] Lustre: lustre-OST0000: new disk, initializing [ 197.627420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.637989] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.654418] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 197.728567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.744568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.768067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.832969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 203.666846] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.322447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 218.458462] Lustre: lustre-OST0001: new disk, initializing [ 218.463457] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 218.472620] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 218.514700] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 224.316202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 224.325698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 224.421609] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.001843] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.791333] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.607529] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 249.994840] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing check_logdir /tmp/testlogs/ [ 255.017707] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing yml_node [ 258.825877] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 261.691217] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 264.481784] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 266.344790] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 16 23:45:03 EDT 2026 [ 283.953790] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 285.536553] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.537190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 301.549476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.560687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.145484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.148401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.161702] Lustre: Skipped 1 previous similar message [ 306.168110] Lustre: Skipped 1 previous similar message [ 307.429726] Lustre: server umount lustre-MDT0000 complete [ 310.720951] LustreError: 10337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789616749 with bad export cookie 17306002470089598134 [ 310.723859] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.730603] LustreError: 10337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 311.276867] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.285443] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.292603] Lustre: Skipped 2 previous similar messages [ 311.301774] Lustre: Skipped 2 previous similar messages [ 315.304104] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 315.313326] Lustre: Skipped 2 previous similar messages [ 320.481132] LustreError: 6522:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.503078] LustreError: 6522:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 321.507523] LustreError: 8982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.536829] LustreError: 8982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 325.088090] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 325.337268] Lustre: server umount lustre-MDT0001 complete [ 329.140547] Lustre: server umount lustre-OST0000 complete [ 332.704872] Lustre: server umount lustre-OST0001 complete [ 338.452070] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 347.751549] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 385.749311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.103726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 386.150955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 386.268540] Lustre: lustre-MDT0000: new disk, initializing [ 386.364513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.391603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 391.179444] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 402.218421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.333226] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 402.389083] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 402.392726] Lustre: Skipped 1 previous similar message [ 402.458259] Lustre: lustre-MDT0001: new disk, initializing [ 402.532035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 402.564623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 402.576520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 407.489420] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 418.152623] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 424.767277] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 432.452249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 432.680363] Lustre: lustre-OST0000: new disk, initializing [ 432.688561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 432.697290] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 432.791232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.647717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 434.663724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 434.833699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 439.786520] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 449.983420] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 456.182652] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 456.511329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 461.635144] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 461.948453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 465.383073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.384123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.392236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.392243] Lustre: Skipped 4 previous similar messages [ 465.426780] Lustre: Skipped 1 previous similar message [ 470.354482] Lustre: server umount lustre-OST0000 complete [ 475.617744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.737656] LustreError: 15081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.750564] LustreError: 15081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 480.793177] Lustre: server umount lustre-MDT0000 complete [ 484.753671] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789616923 with bad export cookie 17306002470089599674 [ 484.763825] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.292965] Lustre: server umount lustre-MDT0001 complete [ 492.852332] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 23:48:49 (1789616929) [ 517.292915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.872111] LustreError: 19923:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.953122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 522.635858] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 523.247782] LustreError: 19924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.903555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.289866] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 544.375984] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 551.553792] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 553.632778] Lustre: Failing over lustre-MDT0000 [ 553.648033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.659664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 553.667017] Lustre: Skipped 2 previous similar messages [ 553.687259] Lustre: Skipped 4 previous similar messages [ 553.884384] Lustre: server umount lustre-MDT0000 complete [ 556.770425] LustreError: 19924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.793462] LustreError: 19924:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 572.545988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.661060] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 577.003229] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 577.004071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 577.036910] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 577.126586] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 586.127986] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 592.353501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 592.372403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.394747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 595.910240] Lustre: server umount lustre-MDT0000 complete [ 598.498269] LustreError: 21133:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.515834] LustreError: 21133:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 599.781849] LustreError: 19904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789617038 with bad export cookie 17306002470089601088 [ 599.786024] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.791541] LustreError: 19904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 600.159768] Lustre: server umount lustre-MDT0001 complete [ 609.431614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.855447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.862801] Lustre: Skipped 2 previous similar messages [ 614.813855] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 624.317649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.093717] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 637.981644] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 644.641320] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 647.027085] Lustre: Failing over lustre-MDT0000 [ 647.314656] Lustre: server umount lustre-MDT0000 complete [ 650.150244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.153783] LustreError: 23257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.164774] Lustre: Skipped 2 previous similar messages [ 650.187587] LustreError: 23257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 665.570419] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789617088/real 1789617088] req@ffff93930fd0e680 x1876548935409536/t0(0) o400->MGC192.168.206.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789617104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 665.609100] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 668.393045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.835321] LustreError: 3644:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff93931ff06680 x1876548935412480/t0(0) o250->MGC192.168.206.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 677.284768] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 682.049262] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 682.471323] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 682.484607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 682.494864] Lustre: Skipped 1 previous similar message [ 682.512600] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 691.860186] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 697.826338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 697.837112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 697.844216] Lustre: Skipped 2 previous similar messages [ 702.394936] Lustre: server umount lustre-MDT0000 complete [ 706.440592] LustreError: 24832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789617144 with bad export cookie 17306002470089602026 [ 706.452407] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.464039] LustreError: 24832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 706.871245] Lustre: server umount lustre-MDT0001 complete [ 716.331096] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 23:52:33 (1789617153) [ 734.574972] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 746.583498] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 792.508576] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 803.919684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 804.196307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 804.253153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 804.352845] Lustre: lustre-MDT0000: new disk, initializing [ 804.423103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.435473] Lustre: Skipped 2 previous similar messages [ 804.451512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 808.636664] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 818.269565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.443039] Lustre: 29638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 818.706252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 823.153958] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 831.940408] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 837.175087] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 844.021534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 844.270166] Lustre: 31605:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 845.701826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 845.740272] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 845.826726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 845.855879] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 849.799889] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 859.857316] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 865.990529] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 886.764297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 886.774381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 886.775296] LustreError: Skipped 1 previous similar message [ 886.778068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 886.799882] Lustre: Skipped 3 previous similar messages [ 886.820598] Lustre: Skipped 3 previous similar messages [ 890.713433] Lustre: server umount lustre-OST0000 complete [ 895.466032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 900.774671] Lustre: server umount lustre-MDT0000 complete [ 902.132706] LustreError: 29650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 902.167331] LustreError: 29650:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 904.921734] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789617343 with bad export cookie 17306002470089602432 [ 904.924607] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 904.928631] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 906.081677] Lustre: server umount lustre-MDT0001 complete [ 923.550234] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 926.766451] Key type lgssc unregistered [ 927.205384] LNet: 33603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.220713] LNetError: 33603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.252627] LNet: Removed LNI 192.168.206.145@tcp [ 928.615134] Key type .llcrypt unregistered [ 928.618626] Key type ._llcrypt unregistered [ 956.675909] Key type ._llcrypt registered [ 956.680488] Key type .llcrypt registered [ 956.804562] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 958.033759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 958.159707] alg: No test for adler32 (adler32-zlib) [ 959.297318] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 959.518153] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 961.211057] Key type lgssc registered [ 962.496688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 969.965834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 969.980486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.409680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.157739] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 980.979543] LustreError: 35014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 980.999733] LustreError: 35014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 984.994201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.296213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 989.869740] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 998.176442] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1004.851791] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1011.733900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1012.073147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.225157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1017.505354] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1026.150361] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1029.501792] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1046.630654] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 23:58:03 (1789617483) [ 1049.217820] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1051.593211] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1060.421800] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 23:58:17 (1789617497) [ 1070.116842] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 23:58:27 (1789617507) [ 1071.359902] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1085.512380] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1085.529525] Lustre: Skipped 2 previous similar messages [ 1087.818286] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 23:58:44 (1789617524) [ 1090.398412] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1090.410271] Lustre: Skipped 2 previous similar messages [ 1099.818426] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1099.829286] Lustre: Skipped 4 previous similar messages [ 1115.188829] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 23:59:12 (1789617552) [ 1118.044298] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1121.755725] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1130.683166] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 23:59:28 (1789617568) [ 1132.738266] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1132.748063] Lustre: Skipped 6 previous similar messages [ 1146.036492] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 23:59:43 (1789617583) [ 1159.650112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1159.657308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.680751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1162.727248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.728771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.729726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.236642] Lustre: server umount lustre-OST0000 complete [ 1169.894410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1169.903634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.912081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.915224] Lustre: Skipped 1 previous similar message [ 1172.967799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1172.970147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.083883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.095791] Lustre: Skipped 2 previous similar messages [ 1183.202245] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1183.363242] Lustre: server umount lustre-MDT0000 complete [ 1186.772638] LustreError: 34995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789617625 with bad export cookie 7357197773030410560 [ 1186.782680] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.790235] LustreError: 34995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1187.129654] Lustre: server umount lustre-MDT0001 complete [ 1199.210833] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 1202.193725] Key type lgssc unregistered [ 1202.524183] LNet: 42874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.537706] LNetError: 42874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.558430] LNet: Removed LNI 192.168.206.145@tcp [ 1203.499965] Key type .llcrypt unregistered [ 1203.505916] Key type ._llcrypt unregistered [ 1221.394488] Key type ._llcrypt registered [ 1221.399672] Key type .llcrypt registered [ 1221.586403] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 1239.889937] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 1241.181185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1241.194729] alg: No test for adler32 (adler32-zlib) [ 1242.334942] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 1242.572115] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 1244.256194] Key type lgssc registered [ 1245.489751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1285.623350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1285.659310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.935103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1286.955385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1287.021616] Lustre: lustre-MDT0000: new disk, initializing [ 1287.114783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.128671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1293.349972] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1305.579710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.794585] Lustre: 46037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1305.866572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1305.873080] Lustre: Skipped 1 previous similar message [ 1305.959686] Lustre: lustre-MDT0001: new disk, initializing [ 1306.099819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1306.141987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1306.159575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1310.752851] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1322.306944] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1328.696838] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1336.694984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1337.024890] Lustre: lustre-OST0000: new disk, initializing [ 1337.029964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1337.040662] Lustre: 48009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1337.153644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1343.839763] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1346.671378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1346.690173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1346.786118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1354.049097] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1359.220548] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1359.415477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1364.727859] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1365.006721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1367.526477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.531085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.532438] LustreError: Skipped 1 previous similar message [ 1367.534991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1367.542198] Lustre: Skipped 1 previous similar message [ 1372.646865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1372.652505] Lustre: Skipped 2 previous similar messages [ 1373.382204] Lustre: server umount lustre-OST0000 complete [ 1382.883969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.884468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1382.885593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.900517] Lustre: Skipped 1 previous similar message [ 1384.245316] Lustre: server umount lustre-MDT0000 complete [ 1389.438454] LustreError: 46031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789617828 with bad export cookie 18327149045285503616 [ 1389.444939] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1389.453706] LustreError: 46031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1390.155102] Lustre: server umount lustre-MDT0001 complete [ 1403.725634] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:04:00 (1789617840) [ 1412.528120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.930929] LustreError: 50469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.943883] LustreError: 50469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1412.986420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.212363] LustreError: 50470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1418.753605] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1423.329980] LustreError: 50469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.396138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.825712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1432.817585] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1441.233572] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1447.584675] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1454.577884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.851820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.105839] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1469.997604] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1478.792162] LustreError: 52998:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1480.027561] Lustre: Modifying parameter general.timeout=1 in log params [ 1481.052963] Lustre: Modifying parameter general.timeout=2 in log params [ 1483.838896] Lustre: Modifying parameter general.timeout=5 in log params [ 1483.855380] Lustre: Skipped 2 previous similar messages [ 1488.602282] Lustre: Modifying parameter general.timeout=10 in log params [ 1488.606249] Lustre: Skipped 4 previous similar messages [ 1496.839333] Lustre: Modifying parameter general.timeout=20 in log params [ 1496.846683] Lustre: Skipped 9 previous similar messages [ 1513.425246] Lustre: Modifying parameter general.timeout=39 in log params [ 1513.437200] Lustre: Skipped 18 previous similar messages [ 1546.363769] Lustre: Modifying parameter general.timeout=71 in log params [ 1546.372318] Lustre: Skipped 31 previous similar messages [ 1611.104680] Lustre: Disabling parameter general.timeout= in log params [ 1611.109617] Lustre: Skipped 58 previous similar messages [ 1614.635700] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:07:31 (1789618051) [ 1617.863544] Lustre: 59954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1619.797859] Lustre: 60055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1621.815789] Lustre: 60153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1625.853861] Lustre: 60362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1625.869380] Lustre: 60362:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1654.329522] Lustre: 61814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1654.345513] Lustre: 61814:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1654.379079] LustreError: 61814:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1659.320938] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:08:16 (1789618096) [ 1673.697407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1673.714070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.737399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.748223] Lustre: Skipped 1 previous similar message [ 1675.765247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.832687] Lustre: server umount lustre-MDT0000 complete [ 1680.865779] LustreError: 51468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.907418] LustreError: 51468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1683.776988] LustreError: 53939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789618122 with bad export cookie 18327149045285504498 [ 1683.781418] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.784370] LustreError: 53939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1684.200699] Lustre: server umount lustre-MDT0001 complete [ 1699.209710] Lustre: server umount lustre-OST0000 complete [ 1724.660090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.909515] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1725.256805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.189540] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1737.021384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.156535] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1737.197571] Lustre: 64347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1741.837712] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1748.985617] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1754.035722] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1759.450570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1759.596846] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1759.815510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.821553] Lustre: Skipped 1 previous similar message [ 1765.640325] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1773.383152] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1775.462067] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1775.468604] Lustre: Skipped 23 previous similar messages [ 1775.530994] Lustre: 66914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1782.753077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1782.759941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.775074] Lustre: Skipped 1 previous similar message [ 1782.784118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.790128] Lustre: Skipped 3 previous similar messages [ 1787.362101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1788.849283] Lustre: server umount lustre-OST0000 complete [ 1792.996315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1793.017711] Lustre: Skipped 1 previous similar message [ 1793.023646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1793.029530] Lustre: Skipped 3 previous similar messages [ 1798.546563] Lustre: server umount lustre-MDT0000 complete [ 1802.106544] LustreError: 64337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789618240 with bad export cookie 18327149045285507564 [ 1802.108971] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1802.123867] LustreError: 64337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1802.386545] Lustre: server umount lustre-MDT0001 complete [ 1813.998511] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 1816.925338] Key type lgssc unregistered [ 1817.150555] LNet: 68122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.167250] LNetError: 68122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.203463] LNet: Removed LNI 192.168.206.145@tcp [ 1818.063843] Key type .llcrypt unregistered [ 1818.066673] Key type ._llcrypt unregistered [ 1839.846601] Key type ._llcrypt registered [ 1839.850821] Key type .llcrypt registered [ 1839.931593] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 1840.899784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1840.916594] alg: No test for adler32 (adler32-zlib) [ 1841.949133] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 1842.208169] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 1843.848207] Key type lgssc registered [ 1844.716861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1851.071498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1851.085304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.427698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.543879] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1861.609879] LustreError: 69530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1861.622647] LustreError: 69530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1864.734479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.467788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1869.585487] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1876.810610] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1882.254958] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1888.298384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1888.669133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.579977] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1899.702793] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1906.597918] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1916.388798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1916.397912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.420058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.413022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.415294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.431692] Lustre: Skipped 1 previous similar message [ 1917.444564] Lustre: Skipped 2 previous similar messages [ 1919.930344] Lustre: server umount lustre-MDT0000 complete [ 1922.530322] LustreError: 69531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.562894] LustreError: 69531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1923.350414] LustreError: 69510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789618361 with bad export cookie 11058974335967278055 [ 1923.358961] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1923.363492] LustreError: 69510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1923.662331] Lustre: server umount lustre-MDT0001 complete [ 1937.382681] Lustre: server umount lustre-OST0000 complete [ 1960.614495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.805852] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1960.827499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1961.040350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.372594] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1971.768520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.870797] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1971.932741] Lustre: 74399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1972.370863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.337567] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 1982.585926] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1987.310441] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1993.213938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1993.340824] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1993.656667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.952280] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2004.957775] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2006.545689] Lustre: 76973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2008.030937] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2015.206277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2015.214767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.223912] LustreError: Skipped 1 previous similar message [ 2015.225629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2015.247861] Lustre: Skipped 1 previous similar message [ 2015.267271] Lustre: Skipped 1 previous similar message [ 2017.612945] Lustre: server umount lustre-OST0000 complete [ 2023.395722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2023.405912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.416871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2030.567278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2030.579663] Lustre: Skipped 3 previous similar messages [ 2035.168956] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2035.293362] Lustre: server umount lustre-MDT0000 complete [ 2038.027048] LustreError: 74391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789618476 with bad export cookie 11058974335967278888 [ 2038.031767] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2038.042535] LustreError: 74391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2038.276678] Lustre: server umount lustre-MDT0001 complete [ 2046.471195] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 2048.578775] Key type lgssc unregistered [ 2048.789574] LNet: 78181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.795576] LNetError: 78181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2048.811516] LNet: Removed LNI 192.168.206.145@tcp [ 2049.459894] Key type .llcrypt unregistered [ 2049.463636] Key type ._llcrypt unregistered [ 2067.825886] Key type ._llcrypt registered [ 2067.828242] Key type .llcrypt registered [ 2067.910394] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2068.509190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2068.573500] alg: No test for adler32 (adler32-zlib) [ 2069.606907] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 2069.816563] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 2071.464239] Key type lgssc registered [ 2072.402125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2079.001761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2079.040876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.415412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.083708] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2089.959692] LustreError: 79569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.974739] LustreError: 79569:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2091.261675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2091.728151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2095.440042] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2101.532895] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2106.878953] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2113.763227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2114.304371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.211176] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2126.697651] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2133.084941] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2133.877878] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2135.503788] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2135.510386] Lustre: Skipped 1 previous similar message [ 2137.645660] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2137.649484] Lustre: Skipped 2 previous similar messages [ 2141.780386] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2141.785943] Lustre: Skipped 5 previous similar messages [ 2150.137379] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2150.140697] Lustre: Skipped 9 previous similar messages [ 2166.602513] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2166.614074] Lustre: Skipped 19 previous similar messages [ 2199.127972] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2199.131795] Lustre: Skipped 40 previous similar messages [ 2221.434268] Lustre: 87660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2224.269472] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:17:41 (1789618661) [ 2225.954589] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2228.258433] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:17:45 (1789618665) [ 2235.369424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2235.379773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.392845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2237.410734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2237.417966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2237.425494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2242.538901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2242.543832] Lustre: Skipped 2 previous similar messages [ 2247.136472] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2247.303175] Lustre: server umount lustre-OST0000 complete [ 2252.771395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.801918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.806189] Lustre: Skipped 1 previous similar message [ 2256.664498] Lustre: server umount lustre-MDT0000 complete [ 2257.892795] LustreError: 80666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.905089] LustreError: 80666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2259.725906] LustreError: 79551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789618698 with bad export cookie 14392252371397752913 [ 2259.727979] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2259.740974] LustreError: 79551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2260.020317] Lustre: server umount lustre-MDT0001 complete [ 2268.976447] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 2271.075433] Key type lgssc unregistered [ 2271.340681] LNet: 88910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.354543] LNetError: 88910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2271.369430] LNet: Removed LNI 192.168.206.145@tcp [ 2272.071852] Key type .llcrypt unregistered [ 2272.075717] Key type ._llcrypt unregistered [ 2275.727029] Key type ._llcrypt registered [ 2275.730478] Key type .llcrypt registered [ 2275.845744] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2279.416416] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules [ 2280.153281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.179795] alg: No test for adler32 (adler32-zlib) [ 2281.251828] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 2281.414853] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 2283.057138] Key type lgssc registered [ 2284.033992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2285.066912] LustreError: 90107:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2301.027564] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2307.552959] LustreError: 90107:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2309.223959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2309.246230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.690759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2314.139682] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2320.870214] LustreError: 91130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.887084] LustreError: 91130:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2321.438450] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:19:19 (1789618759) [ 2325.340805] LustreError: 91162:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2325.350887] LustreError: 91162:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2325.360799] LustreError: 91162:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2325.588350] Lustre: server umount lustre-MDT0000 complete [ 2337.397779] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 2339.830530] Key type lgssc unregistered [ 2340.088507] LNet: 92655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.096247] LNetError: 92655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2340.113624] LNet: Removed LNI 192.168.206.145@tcp [ 2340.794447] Key type .llcrypt unregistered [ 2340.796457] Key type ._llcrypt unregistered [ 2356.970991] Key type ._llcrypt registered [ 2356.974441] Key type .llcrypt registered [ 2357.136672] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2358.111597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2358.149023] alg: No test for adler32 (adler32-zlib) [ 2359.175902] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 2359.387622] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 2361.066530] Key type lgssc registered [ 2362.146340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2370.253786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2370.271602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.781852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.241842] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2382.305280] LustreError: 93873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.346696] LustreError: 93873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2386.059504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.667592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2390.440707] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2396.162954] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2400.658658] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2407.148984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2407.897564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.079125] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2421.297210] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2465.421340] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:21:42 (1789618902) [ 2467.083205] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2468.785230] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:21:46 (1789618906) [ 2473.442888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2473.450532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.507137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.523329] Lustre: Skipped 2 previous similar messages [ 2478.563299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2478.573784] Lustre: Skipped 2 previous similar messages [ 2479.330951] Lustre: server umount lustre-MDT0000 complete [ 2482.632190] LustreError: 93853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789618921 with bad export cookie 10947988060719869025 [ 2482.634178] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2482.659254] LustreError: 93853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2483.086599] Lustre: server umount lustre-MDT0001 complete [ 2497.162704] Lustre: server umount lustre-OST0000 complete [ 2509.899720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.333904] LustreError: 98159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.492526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.097742] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2515.939177] LustreError: 98160:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.041973] LustreError: 98159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.196146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.800378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2527.310767] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2533.954773] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2538.738631] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2550.497780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2550.702355] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2550.708975] LustreError: 98147:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2550.725812] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2550.732651] LustreError: 100241:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2550.737983] LustreError: 100241:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2550.743149] LustreError: 100241:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2550.802549] Lustre: server umount lustre-OST0000 complete [ 2550.807285] LustreError: 100241:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2557.932335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2558.061160] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2558.066507] LustreError: 98146:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2558.071778] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2558.076517] LustreError: 100706:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2558.080941] LustreError: 100706:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2558.084822] LustreError: 100706:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2558.135544] Lustre: server umount lustre-OST0000 complete [ 2558.138175] LustreError: 100706:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2566.744917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2566.951931] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2566.958811] Lustre: Found index 0 for lustre-OST0000, updating log [ 2566.970937] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2567.030807] Lustre: lustre-OST0000: new disk, initializing [ 2567.033825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2567.041484] Lustre: 101240:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2567.254921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.005933] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2575.175950] Lustre: Failing over lustre-OST0000 [ 2575.330675] Lustre: server umount lustre-OST0000 complete [ 2579.937848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2579.940626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.957322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2579.981671] Lustre: Skipped 3 previous similar messages [ 2584.595421] Lustre: server umount lustre-MDT0000 complete [ 2585.060708] LustreError: 98890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.085198] LustreError: 98890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2588.074517] LustreError: 98900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789619026 with bad export cookie 10947988060719871762 [ 2588.079327] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2588.082533] LustreError: 98900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2598.549266] Lustre: server umount lustre-MDT0001 complete [ 2606.961663] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:24:04 (1789619044) [ 2622.791801] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2631.567757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.900336] LustreError: 103585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.014396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.877300] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2643.884736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2648.522305] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2651.173630] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2657.287444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2658.163174] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2663.705831] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2669.418489] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2669.434689] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2669.435524] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2670.738972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2670.789548] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2670.860388] Lustre: lustre-OST0001: new disk, initializing [ 2670.863988] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2670.869643] Lustre: 105903:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2671.126351] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2671.133164] Lustre: Skipped 2 previous similar messages [ 2675.789337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2675.796030] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2675.905986] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2676.439289] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2683.683547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2691.134984] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2695.649923] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2695.653802] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.661732] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2695.665793] Lustre: Skipped 1 previous similar message [ 2697.711561] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.727820] Lustre: Skipped 1 previous similar message [ 2701.497842] Lustre: server umount lustre-MDT0001 complete [ 2702.837189] LustreError: 103586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.861270] LustreError: 103586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2710.058421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.126329] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2710.130776] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2710.163680] Lustre: 103574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2712.123729] LustreError: 107494:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2712.135103] LustreError: 107494:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2712.150269] LustreError: 103578:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2712.252865] LustreError: 107505:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.145@tcp: cfg command failed: rc = -17 [ 2712.261800] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2712.261800] [ 2714.600650] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2716.150991] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2723.927417] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:26:01 (1789619161) [ 2739.583258] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2754.858603] Lustre: 109517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2775.527583] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2779.062100] Lustre: 110652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2864.100892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2864.127197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.140173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2864.150296] Lustre: Skipped 4 previous similar messages [ 2866.845970] Lustre: server umount lustre-MDT0000 complete [ 2869.729293] LustreError: 103582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.756269] LustreError: 103582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2870.699592] LustreError: 103567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789619309 with bad export cookie 10947988060719872700 [ 2870.710509] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.723097] LustreError: 103567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2871.119989] Lustre: server umount lustre-MDT0001 complete [ 2885.413666] Lustre: server umount lustre-OST0000 complete [ 2899.304830] Lustre: server umount lustre-OST0001 complete [ 2905.974630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2916.175835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2925.301828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2934.877172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2953.921780] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 2963.903791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.932409] Lustre: lustre-MDT0000: reset Object Index mappings [ 2964.329948] LustreError: 114604:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2964.463664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.469480] Lustre: Skipped 1 previous similar message [ 2968.438623] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2975.929667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.950583] Lustre: lustre-MDT0001: reset Object Index mappings [ 2980.788961] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 2983.267908] Lustre: 115750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2989.319531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2994.905365] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3001.288434] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3002.213804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3008.001808] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3008.558090] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3009.018057] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3009.026245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3017.369577] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3020.960162] Lustre: 117627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3071.898392] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:31:49 (1789619509) [ 3099.107867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3099.121221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.138107] Lustre: Skipped 3 previous similar messages [ 3099.143240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3099.148573] Lustre: Skipped 3 previous similar messages [ 3104.264794] Lustre: server umount lustre-MDT0000 complete [ 3106.277599] LustreError: 117641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.298796] LustreError: 117641:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3108.048038] LustreError: 114586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789619546 with bad export cookie 10947988060719919551 [ 3108.054502] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3108.065975] LustreError: 114586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3114.522043] Lustre: server umount lustre-MDT0001 complete [ 3123.923279] Lustre: server umount lustre-OST0000 complete [ 3132.843490] Lustre: server umount lustre-OST0001 complete [ 3138.203516] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 3147.223442] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 3178.710670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.970740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3179.021665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3179.112250] Lustre: lustre-MDT0000: new disk, initializing [ 3179.210409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3179.215376] Lustre: Skipped 3 previous similar messages [ 3179.238627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3183.055616] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3189.412707] Lustre: server umount lustre-MDT0000 complete [ 3197.992768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.302292] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3198.311530] Lustre: 121895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3198.322745] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3203.074867] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3207.619864] Lustre: server umount lustre-MDT0000 complete [ 3216.123717] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:34:13 (1789619653) [ 3217.659286] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3219.237343] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:34:16 (1789619656) [ 3235.936396] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 3245.229823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3249.352356] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3256.768519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.863773] Lustre: 123943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3256.913161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3256.917028] Lustre: Skipped 1 previous similar message [ 3256.986702] Lustre: lustre-MDT0001: new disk, initializing [ 3257.112940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3257.122573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3260.826529] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3265.165562] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3271.333825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3271.597279] Lustre: lustre-OST0000: new disk, initializing [ 3271.609982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3271.641548] Lustre: 125520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3272.805718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3272.818437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3272.929328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3276.556526] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3285.329909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.454175] Lustre: lustre-OST0001: new disk, initializing [ 3285.457158] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3285.461891] Lustre: 126389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3287.008103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3287.019661] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3287.104517] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3290.304339] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 3299.013969] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3302.105976] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3643.227134] Lustre: 132798:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3643.234412] Lustre: 132798:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3643.243207] Lustre: 132798:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3643.250309] Lustre: 132798:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3643.257411] Lustre: 132798:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3643.267127] Lustre: 132798:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3914.546081] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:45:51 (1789620351) [ 3932.131520] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.150522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3932.160240] Lustre: Skipped 7 previous similar messages [ 3932.178065] Lustre: Skipped 11 previous similar messages [ 3935.017333] Lustre: server umount lustre-MDT0000 complete [ 3937.251574] LustreError: 127232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.275403] LustreError: 127232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 3938.874840] LustreError: 123934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789620377 with bad export cookie 10947988060719987780 [ 3938.887372] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3938.894074] LustreError: 123934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3942.370330] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.380153] Lustre: Skipped 1 previous similar message [ 3942.384497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3942.394920] Lustre: Skipped 1 previous similar message [ 3945.668257] Lustre: server umount lustre-MDT0001 complete [ 3963.360174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3963.453672] Lustre: server umount lustre-OST0000 complete [ 3981.280162] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3981.470883] Lustre: server umount lustre-OST0001 complete [ 3987.685618] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 3996.397242] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 4035.996120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4036.266148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4036.315806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4036.413450] Lustre: lustre-MDT0000: new disk, initializing [ 4036.509345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4036.516413] Lustre: Skipped 5 previous similar messages [ 4036.532272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4041.228800] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4051.298036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.424180] Lustre: 135848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4051.492091] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4051.498504] Lustre: Skipped 1 previous similar message [ 4051.586213] Lustre: lustre-MDT0001: new disk, initializing [ 4051.728475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4051.737796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4056.907427] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4066.329871] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4072.584158] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4079.334983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4079.621505] Lustre: lustre-OST0000: new disk, initializing [ 4079.633315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4079.643174] Lustre: 137816:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4081.042960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4081.056513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4081.137204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4086.123932] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 4095.621304] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4105.474725] Lustre: lustre-MDD0000: changelog on [ 4108.076599] Lustre: lustre-MDD0001: changelog on [ 5827.553479] Lustre: lustre-MDD0001: changelog off [ 5829.286207] Lustre: 135855:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5830.328860] Lustre: lustre-MDD0000: changelog off [ 5836.776835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5836.782414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5836.795865] Lustre: Skipped 1 previous similar message [ 5836.803577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5836.808379] Lustre: Skipped 2 previous similar messages [ 5837.289720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.478957] Lustre: server umount lustre-OST0000 complete [ 5847.522380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.526183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5847.534741] Lustre: Skipped 1 previous similar message [ 5847.549621] Lustre: Skipped 3 previous similar messages [ 5850.639134] Lustre: server umount lustre-MDT0000 complete [ 5852.641371] LustreError: 135859:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.657772] LustreError: 135859:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5853.595440] LustreError: 135842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789622292 with bad export cookie 10947988060720806276 [ 5853.598091] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5853.610113] LustreError: 135842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5853.871930] Lustre: server umount lustre-MDT0001 complete [ 5862.354401] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 5864.245153] Key type lgssc unregistered [ 5864.441848] LNet: 146931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5864.448056] LNetError: 146931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5864.462880] LNet: Removed LNI 192.168.206.145@tcp [ 5865.101125] Key type .llcrypt unregistered [ 5865.106483] Key type ._llcrypt unregistered [ 5879.813393] Key type ._llcrypt registered [ 5879.815618] Key type .llcrypt registered [ 5879.914474] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 5891.956978] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 5892.712808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5892.820666] alg: No test for adler32 (adler32-zlib) [ 5893.869993] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 5894.096975] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 5895.744153] Key type lgssc registered [ 5896.595258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5923.734183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5923.762228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5924.974894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5925.000703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5925.099734] Lustre: lustre-MDT0000: new disk, initializing [ 5925.178469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5925.197307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5928.729096] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5937.359242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5937.451792] Lustre: 150106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5937.501547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5937.504792] Lustre: Skipped 1 previous similar message [ 5937.563470] Lustre: lustre-MDT0001: new disk, initializing [ 5937.642486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5937.677494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5937.690931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5941.311189] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5948.630941] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5953.086849] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5958.544853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5958.774676] Lustre: lustre-OST0000: new disk, initializing [ 5958.783898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5958.797080] Lustre: 152072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5958.872663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5963.361370] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 5967.397042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5967.412947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5967.491796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5970.781698] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5974.877926] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_module obdecho/obdecho [ 5976.171244] LustreError: 153073:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5976.178626] LustreError: 153073:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5977.474424] LustreError: 153170:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5988.324553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.326827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5988.338655] Lustre: Skipped 2 previous similar messages [ 5988.342981] Lustre: Skipped 1 previous similar message [ 5992.668909] Lustre: server umount lustre-MDT0000 complete [ 5994.944535] LustreError: 150899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789622433 with bad export cookie 8856944534976755732 [ 5994.947816] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5994.956526] LustreError: 150899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5995.194471] Lustre: server umount lustre-MDT0001 complete [ 6008.153734] Lustre: server umount lustre-OST0000 complete [ 6023.057903] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6030.950548] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6061.499050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.667258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6061.687284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6061.741524] Lustre: lustre-MDT0000: new disk, initializing [ 6061.810304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6061.826927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6065.607587] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6075.560166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6075.681513] Lustre: 156884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6075.750169] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6075.755926] Lustre: Skipped 1 previous similar message [ 6075.875137] Lustre: lustre-MDT0001: new disk, initializing [ 6075.957582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6075.989790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6076.004584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6080.007069] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6088.606984] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6093.431715] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6100.119431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6100.383811] Lustre: lustre-OST0000: new disk, initializing [ 6100.387761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6100.392481] Lustre: 158851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6100.450924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6105.704410] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6107.689529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6107.704523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6107.796084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6113.831651] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6117.551844] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6117.806406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6121.314897] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6121.509870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6122.984579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.987185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6122.996371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6123.001680] Lustre: Skipped 2 previous similar messages [ 6128.109125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6128.123364] Lustre: Skipped 1 previous similar message [ 6133.224893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6133.235581] Lustre: Skipped 1 previous similar message [ 6137.312087] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6137.468228] Lustre: server umount lustre-OST0000 complete [ 6142.433243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6142.437064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.446964] Lustre: Skipped 1 previous similar message [ 6142.458478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6146.301701] Lustre: server umount lustre-MDT0000 complete [ 6148.579885] LustreError: 158006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.598934] LustreError: 158006:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6149.194094] LustreError: 156876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789622587 with bad export cookie 8856944534976756684 [ 6149.203841] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6149.457883] Lustre: server umount lustre-MDT0001 complete [ 6152.460535] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:23:10 (1789622590) [ 6153.682487] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6155.013327] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:23:12 (1789622592) [ 6160.952039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6161.301032] LustreError: 161264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.384622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6164.724858] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6166.498493] LustreError: 161265:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6170.603573] LustreError: 161264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6171.546612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.403425] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6181.978652] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6186.818507] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6192.160363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6197.643959] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6204.065419] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6215.335441] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.45@tcp (stopping) [ 6215.342883] Lustre: Skipped 2 previous similar messages [ 6216.167955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6216.187910] Lustre: Skipped 2 previous similar messages [ 6217.475922] Lustre: server umount lustre-MDT0000 complete [ 6220.426996] LustreError: 161261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6220.522931] LustreError: 161245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789622659 with bad export cookie 8856944534976757566 [ 6220.528263] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6220.532655] LustreError: 161245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6221.282663] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6221.294730] Lustre: Skipped 1 previous similar message [ 6226.826451] Lustre: server umount lustre-MDT0001 complete [ 6231.649459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6240.843281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.022790] LustreError: 165103:0:(llog.c:1655:llog_backup()) MGC192.168.206.145@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6241.030281] Lustre: 165103:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.145@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6246.089379] LustreError: 165108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6246.102617] LustreError: 165108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6262.413357] LustreError: 165109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.422381] LustreError: 165109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 6297.058510] LustreError: 165108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.069852] LustreError: 165108:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 6302.176883] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6302.184037] Lustre: 165103:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6342.705304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6342.715799] Lustre: Skipped 2 previous similar messages [ 6345.713901] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6347.755834] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6351.217280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6362.762341] LustreError: 165856:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6362.778945] LustreError: 165856:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 6367.208680] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7aea3020e2a0989e [ 6367.219175] Lustre: MGC192.168.206.145@tcp: Connection restored to 0@lo (at 0@lo) [ 6370.034980] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6372.846239] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6374.785895] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6375.819836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6379.697885] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6380.912321] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6397.266689] Lustre: server umount lustre-MDT0000 complete [ 6398.437143] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6398.446335] Lustre: Skipped 1 previous similar message [ 6398.447655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6399.978516] LustreError: 162003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789622838 with bad export cookie 8856944534976764062 [ 6399.982733] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6399.985547] LustreError: 162003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6400.223583] Lustre: server umount lustre-MDT0001 complete [ 6413.378785] Lustre: server umount lustre-OST0000 complete [ 6427.170472] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6434.215304] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6458.306736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6458.436657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6458.455933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6458.501642] Lustre: lustre-MDT0000: new disk, initializing [ 6458.546927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6458.552946] Lustre: Skipped 1 previous similar message [ 6458.562356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6461.336380] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6468.568650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6468.643895] Lustre: 170203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6468.800186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6471.397185] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6477.401299] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6480.996923] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6485.710871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6485.875901] Lustre: 172166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6489.624538] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6495.266514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6495.276757] Lustre: Skipped 1 previous similar message [ 6495.279842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6495.340945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6495.686930] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6498.617483] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6498.764272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6502.089952] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6502.248149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6505.448670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6505.448717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6505.455529] LustreError: Skipped 1 previous similar message [ 6505.461308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6505.468356] Lustre: Skipped 2 previous similar messages [ 6505.472025] Lustre: Skipped 6 previous similar messages [ 6509.837637] Lustre: server umount lustre-OST0000 complete [ 6515.170465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6518.629670] Lustre: server umount lustre-MDT0000 complete [ 6520.806827] LustreError: 172623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.833689] LustreError: 172623:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6521.132069] LustreError: 170198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789622959 with bad export cookie 8856944534976780869 [ 6521.139932] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6521.140899] LustreError: 170198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6521.343135] Lustre: server umount lustre-MDT0001 complete [ 6527.231523] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:29:25 (1789622965) [ 6531.615284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.920802] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6540.226947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6543.030588] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6547.474411] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6550.853794] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6555.296420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6559.654209] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6564.320545] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6574.417266] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:30:12 (1789623012) [ 6577.121880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6577.127828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6577.136358] Lustre: Skipped 2 previous similar messages [ 6577.141881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6577.146631] Lustre: Skipped 3 previous similar messages [ 6582.439587] Lustre: server umount lustre-OST0000 complete [ 6590.531469] Lustre: server umount lustre-MDT0000 complete [ 6592.550462] LustreError: 174560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789623031 with bad export cookie 8856944534976781751 [ 6592.556782] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6592.557655] LustreError: 174560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6592.757355] Lustre: server umount lustre-MDT0001 complete [ 6599.362701] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6601.039643] Key type lgssc unregistered [ 6601.193709] LNet: 178300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6601.199731] LNetError: 178300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6601.219929] LNet: Removed LNI 192.168.206.145@tcp [ 6601.728978] Key type .llcrypt unregistered [ 6601.731504] Key type ._llcrypt unregistered [ 6613.507323] Key type ._llcrypt registered [ 6613.509677] Key type .llcrypt registered [ 6613.575217] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6614.084992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6614.152854] alg: No test for adler32 (adler32-zlib) [ 6615.091693] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 6615.237543] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 6616.864401] Key type lgssc registered [ 6617.549179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6622.106845] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6622.119286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.601551] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6642.720438] LustreError: 179516:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.145@tcp: failed processing log, type 4: rc = -110 [ 6647.840622] LustreError: 179516:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6647.846634] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6647.848982] LustreError: MGC192.168.206.145@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6647.854243] LustreError: 179516:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6647.857428] LustreError: 179516:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 6647.862281] LustreError: 179516:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6647.891362] Lustre: server umount lustre-OST0000 complete [ 6647.895264] LustreError: 179516:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6652.647641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6653.875673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6655.996896] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6660.282572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6660.464367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6662.369961] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6666.318526] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6669.163286] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6672.698608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6672.825077] LustreError: 181832:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6672.831221] Lustre: 181832:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.145@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6672.902846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6676.081254] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6679.576894] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6681.059237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6681.063177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.069745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6684.131578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6684.132126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6684.138286] Lustre: Skipped 1 previous similar message [ 6684.150945] Lustre: Skipped 1 previous similar message [ 6685.156704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6685.160048] Lustre: Skipped 2 previous similar messages [ 6686.871228] Lustre: server umount lustre-MDT0000 complete [ 6688.671628] LustreError: 179996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789623127 with bad export cookie 13423232941859961877 [ 6688.673730] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6688.678551] LustreError: 179996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6689.208265] LustreError: 180745:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000adefc32e x1876555782446336/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6689.294628] Lustre: server umount lustre-MDT0001 complete [ 6700.617375] Lustre: server umount lustre-OST0000 complete [ 6705.234647] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6710.038458] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6729.044687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.181696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6729.211839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6729.262281] Lustre: lustre-MDT0000: new disk, initializing [ 6729.318702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6729.329452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6731.706553] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6737.351885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.409218] Lustre: 185350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6737.451219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6737.456968] Lustre: Skipped 1 previous similar message [ 6737.510177] Lustre: lustre-MDT0001: new disk, initializing [ 6737.564578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6737.593732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6737.600298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6739.657672] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6744.916658] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6747.895424] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6751.289686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6751.460196] Lustre: lustre-OST0000: new disk, initializing [ 6751.466603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6751.472487] Lustre: 187317:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6751.516895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6754.667856] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6760.176183] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6760.457639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6760.466531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6760.514880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6762.601373] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6762.722169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6765.260920] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6765.391768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6770.662843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6770.663515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6770.671464] LustreError: Skipped 1 previous similar message [ 6770.672220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6770.678172] Lustre: Skipped 1 previous similar message [ 6770.690814] Lustre: Skipped 1 previous similar message [ 6772.611858] Lustre: server umount lustre-OST0000 complete [ 6775.778234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.784934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6775.791323] Lustre: Skipped 1 previous similar message [ 6780.884460] Lustre: server umount lustre-MDT0000 complete [ 6780.897267] LustreError: 186153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6780.903720] LustreError: 186153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6783.087950] LustreError: 185344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789623221 with bad export cookie 13423232941859962598 [ 6783.089560] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6783.095440] LustreError: 185344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6783.261321] Lustre: server umount lustre-MDT0001 complete [ 6787.991492] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:33:45 (1789623225) [ 6797.535159] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 6799.166561] Key type lgssc unregistered [ 6799.349631] LNet: 190259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6799.354606] LNetError: 190259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6799.364602] LNet: Removed LNI 192.168.206.145@tcp [ 6799.852287] Key type .llcrypt unregistered [ 6799.854186] Key type ._llcrypt unregistered [ 6810.588942] Key type ._llcrypt registered [ 6810.591440] Key type .llcrypt registered [ 6810.667154] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6811.180806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6811.194598] alg: No test for adler32 (adler32-zlib) [ 6812.156622] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 6812.345821] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 6813.968228] Key type lgssc registered [ 6814.711377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6819.668370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6819.678491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6820.967981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6823.637407] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6829.638362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.895635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6833.066895] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6838.187130] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6841.607148] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6846.190558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6846.347359] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6846.352448] LustreError: 193312:0:(llog.c:1696:llog_backup()) MGC192.168.206.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6846.358978] Lustre: 193312:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.145@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6846.436831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6849.994142] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6854.733313] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6859.746218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.748909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6859.759721] Lustre: Skipped 2 previous similar messages [ 6859.771477] Lustre: Skipped 2 previous similar messages [ 6862.222387] Lustre: server umount lustre-MDT0000 complete [ 6864.460322] LustreError: 191430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789623303 with bad export cookie 3999479960219746472 [ 6864.464782] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6864.466987] LustreError: 191430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6864.618197] Lustre: server umount lustre-MDT0001 complete [ 6877.123042] Lustre: server umount lustre-OST0000 complete [ 6882.443256] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 6887.290302] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6906.998769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6907.150603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6907.174276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6907.223304] Lustre: lustre-MDT0000: new disk, initializing [ 6907.270809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6907.284578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6909.902525] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6916.775654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.841523] Lustre: 196830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6916.871975] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6916.874024] Lustre: Skipped 1 previous similar message [ 6916.876791] LustreError: 197611:0:(llog.c:1696:llog_backup()) MGC192.168.206.145@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6916.881753] LustreError: 197611:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6916.881993] LustreError: 196840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6916.885442] Lustre: 197611:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.145@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6916.893048] LustreError: 196840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6916.910243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6916.912943] Lustre: Skipped 1 previous similar message [ 6916.958498] Lustre: lustre-MDT0001: new disk, initializing [ 6917.024398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6917.051900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6917.058627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6919.386199] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6925.094875] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6928.546516] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6933.150488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6933.307694] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6933.309963] Lustre: Skipped 1 previous similar message [ 6933.313836] LustreError: 198771:0:(llog.c:1696:llog_backup()) MGC192.168.206.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6933.319855] LustreError: 198771:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6933.323470] Lustre: 198771:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.145@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6933.330763] Lustre: 198771:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6933.346307] Lustre: lustre-OST0000: new disk, initializing [ 6933.350106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6933.357213] Lustre: 198797:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6933.408372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6933.647363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6933.654517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6933.723545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6936.962313] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6942.383529] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6944.736748] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6944.854288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6947.311111] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6947.418655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6949.346511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6949.347115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6949.354635] LustreError: Skipped 1 previous similar message [ 6949.362655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6949.367623] Lustre: Skipped 1 previous similar message [ 6954.467160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6954.474799] Lustre: Skipped 1 previous similar message [ 6954.689375] Lustre: server umount lustre-OST0000 complete [ 6958.049833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6958.057181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6958.065020] Lustre: Skipped 1 previous similar message [ 6958.068328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6964.706207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6964.709731] Lustre: Skipped 3 previous similar messages [ 6971.360442] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6971.471463] Lustre: server umount lustre-MDT0000 complete [ 6973.726993] LustreError: 197624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789623412 with bad export cookie 3999479960219747186 [ 6973.731031] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6973.736888] LustreError: 197624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6973.943222] Lustre: server umount lustre-MDT0001 complete [ 6978.676114] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:36:56 (1789623416) [ 6987.300362] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 6992.935507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6993.196105] LustreError: 201921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.220262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6995.527875] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 6998.500194] LustreError: 201922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7000.396959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7002.837449] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7004.300392] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7007.857258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7011.399534] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7015.770161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7015.857254] Lustre: lustre-OST0001: new disk, initializing [ 7015.859463] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7015.862855] Lustre: 204226:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7017.432463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7017.439548] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7017.501207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7019.077234] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7026.052878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7029.582308] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7030.718704] Lustre: DEBUG MARKER: ADD OST3 [ 7033.315925] loop: module loaded [ 7033.353456] loop0: detected capacity change from 0 to 204800000 [ 7033.488559] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7033.496817] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7033.504975] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7036.880068] loop0: detected capacity change from 0 to 204800000 [ 7039.023167] Lustre: DEBUG MARKER: STOP OST3 [ 7039.891033] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7040.023651] Lustre: lustre-OST0003: new disk, initializing [ 7040.028764] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7040.033813] Lustre: 206093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7040.125523] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7040.129627] Lustre: Skipped 3 previous similar messages [ 7041.613112] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7041.667904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7041.674978] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7041.703828] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7042.119135] Lustre: Failing over lustre-OST0003 [ 7042.203187] Lustre: server umount lustre-OST0003 complete [ 7043.270893] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7043.681120] LustreError: 206350:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7046.823949] LustreError: 204407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7047.142185] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7047.144386] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7047.153196] Lustre: Skipped 1 previous similar message [ 7048.709729] Lustre: DEBUG MARKER: START OST3 again [ 7051.913812] LustreError: 203406:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.924359] LustreError: 203406:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7052.661494] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7052.806971] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7054.105722] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7054.648428] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7054.648488] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7056.052061] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7071.813351] Lustre: Failing over lustre-OST0003 [ 7071.874822] Lustre: server umount lustre-OST0003 complete [ 7072.418190] LustreError: 203408:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.429770] LustreError: 203408:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7073.260166] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7073.269986] Lustre: Skipped 1 previous similar message [ 7087.586820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7087.590560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.599761] Lustre: Skipped 1 previous similar message [ 7087.604841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7087.607519] Lustre: Skipped 2 previous similar messages [ 7088.615274] LustreError: 203408:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.628883] LustreError: 203408:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 7100.384115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7100.491555] Lustre: server umount lustre-MDT0000 complete [ 7102.371807] LustreError: 204997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789623540 with bad export cookie 3999479960219748068 [ 7102.374981] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7102.381051] LustreError: 204997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7103.971738] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.974059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7103.978852] Lustre: Skipped 4 previous similar messages [ 7103.985804] Lustre: Skipped 13 previous similar messages [ 7108.657797] Lustre: server umount lustre-MDT0001 complete [ 7110.551703] Lustre: server umount lustre-OST0000 complete [ 7112.582817] Lustre: server umount lustre-OST0001 complete [ 7115.504605] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 7119.384875] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7137.793664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.923269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7137.957217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7138.005264] Lustre: lustre-MDT0000: new disk, initializing [ 7138.043096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7138.047485] Lustre: Skipped 1 previous similar message [ 7138.062243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7140.337709] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7145.981749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7146.044706] Lustre: 211529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7146.182642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7148.091097] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7153.365087] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7156.093766] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7159.483028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7159.619100] Lustre: lustre-OST0000: new disk, initializing [ 7159.621664] Lustre: Skipped 1 previous similar message [ 7159.623902] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7159.627057] Lustre: Skipped 2 previous similar messages [ 7159.629857] Lustre: 213495:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7160.843382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7160.849908] Lustre: Skipped 1 previous similar message [ 7160.853189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7160.905073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7162.667334] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7167.779556] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7169.741648] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7169.856440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7171.771634] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7171.898182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7176.163269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7176.163680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.166365] LustreError: Skipped 1 previous similar message [ 7176.167258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.170357] Lustre: Skipped 1 previous similar message [ 7176.173735] Lustre: Skipped 2 previous similar messages [ 7179.048349] Lustre: server umount lustre-OST0000 complete [ 7187.074746] Lustre: server umount lustre-MDT0000 complete [ 7188.869950] LustreError: 211522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789623627 with bad export cookie 3999479960219750364 [ 7188.871396] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7188.879516] LustreError: 211522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7189.076154] Lustre: server umount lustre-MDT0001 complete [ 7190.839581] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:40:29 (1789623629) [ 7198.647315] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 7203.609189] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7220.777506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7220.874296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7220.887117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7220.927633] Lustre: lustre-MDT0000: new disk, initializing [ 7220.983072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7223.095572] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7228.929075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.988117] Lustre: 218097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7229.144037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7231.219095] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7236.241326] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7238.964510] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7242.334722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7242.495396] Lustre: 220066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7243.674462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7245.565546] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7250.639903] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7252.748470] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7252.853996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7254.791648] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7254.886908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7259.110796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7259.112287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7259.116069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7259.116077] Lustre: Skipped 5 previous similar messages [ 7259.130026] Lustre: Skipped 3 previous similar messages [ 7270.371567] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7270.467699] Lustre: server umount lustre-OST0000 complete [ 7278.655925] Lustre: server umount lustre-MDT0000 complete [ 7279.585124] LustreError: 218110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.597770] LustreError: 218110:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7280.555547] LustreError: 218091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789623719 with bad export cookie 3999479960219751239 [ 7280.558895] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7280.561519] LustreError: 218091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7280.718839] Lustre: server umount lustre-MDT0001 complete [ 7285.089094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7285.388167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7285.392940] Lustre: Skipped 5 previous similar messages [ 7287.478989] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7292.202398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7294.388476] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7297.950661] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7300.666613] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7303.707837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7306.365722] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7309.500883] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7322.595187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7322.600163] LustreError: Skipped 1 previous similar message [ 7324.327152] Lustre: server umount lustre-OST0000 complete [ 7326.394699] Lustre: server umount lustre-MDT0000 complete [ 7328.368771] LustreError: 222310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789623766 with bad export cookie 3999479960219752128 [ 7328.373462] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7328.373763] LustreError: 222310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7328.534822] Lustre: server umount lustre-MDT0001 complete [ 7442.051356] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 7443.310172] Key type lgssc unregistered [ 7443.454488] LNet: 225965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7443.460037] LNetError: 225965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7443.469535] LNet: Removed LNI 192.168.206.145@tcp [ 7443.906399] Key type .llcrypt unregistered [ 7443.909697] Key type ._llcrypt unregistered [ 7454.023316] Key type ._llcrypt registered [ 7454.024632] Key type .llcrypt registered [ 7454.067126] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 7461.392352] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7461.739358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7461.768447] alg: No test for adler32 (adler32-zlib) [ 7462.653794] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 7462.770595] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 7464.368188] Key type lgssc registered [ 7464.871667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7479.846534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7479.856592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7480.963925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7480.977408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7481.015532] Lustre: lustre-MDT0000: new disk, initializing [ 7481.048372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7481.056514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7482.746751] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7487.795308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7487.846852] Lustre: 229286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7487.874748] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7487.876541] Lustre: Skipped 1 previous similar message [ 7487.914167] Lustre: lustre-MDT0001: new disk, initializing [ 7487.954089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7487.972558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7487.976911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7489.702529] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7493.884336] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7496.038962] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7498.601433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7498.720588] Lustre: lustre-OST0000: new disk, initializing [ 7498.722906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7498.726910] Lustre: 231255:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7498.759202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7501.054478] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7504.904655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7504.910664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7504.960612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7505.525718] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7507.272471] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7507.367570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7509.138297] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7509.248461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7515.109765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7515.111483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7515.111738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7515.113822] Lustre: Skipped 1 previous similar message [ 7520.226570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7520.229464] Lustre: Skipped 2 previous similar messages [ 7524.320508] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7524.389878] Lustre: server umount lustre-OST0000 complete [ 7528.929290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7528.934620] LustreError: Skipped 1 previous similar message [ 7528.936418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7528.943878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7530.465595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7532.136575] Lustre: server umount lustre-MDT0000 complete [ 7533.680199] LustreError: 229278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789623972 with bad export cookie 16027819962779133596 [ 7533.681623] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7533.686423] LustreError: 229278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7533.807694] Lustre: server umount lustre-MDT0001 complete [ 7540.943406] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7545.057127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7545.293086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7547.001870] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7550.433549] LustreError: 234188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.439758] LustreError: 234188:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7550.624789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7550.788964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7552.673105] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7554.078197] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7556.880580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7559.668098] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7563.576616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7563.657906] Lustre: lustre-OST0001: new disk, initializing [ 7563.662226] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7563.666450] Lustre: 236496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7563.714451] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7563.717365] Lustre: Skipped 1 previous similar message [ 7566.277867] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7568.411184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7568.416131] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7568.459167] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7572.564207] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7575.653982] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7580.594442] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:46:58 (1789624018) [ 7581.665382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7581.668167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7581.673492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7581.675918] Lustre: Skipped 2 previous similar messages [ 7587.505728] Lustre: server umount lustre-MDT0000 complete [ 7589.033245] LustreError: 235694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789624027 with bad export cookie 16027819962779134464 [ 7589.037427] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7589.038253] LustreError: 235694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7589.168912] Lustre: server umount lustre-MDT0001 complete [ 7600.965983] Lustre: server umount lustre-OST0000 complete [ 7611.695744] Lustre: server umount lustre-OST0001 complete [ 7614.356277] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 7618.372195] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7632.507364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7632.601730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7632.616120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7632.654952] Lustre: lustre-MDT0000: new disk, initializing [ 7632.678510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7632.685039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7634.251785] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7638.745770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7638.796359] Lustre: 240435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7638.919464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7640.422727] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7644.577315] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7646.760820] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7649.508117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7649.637613] Lustre: lustre-OST0000: new disk, initializing [ 7649.639759] Lustre: Skipped 1 previous similar message [ 7649.642168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7649.644536] Lustre: Skipped 2 previous similar messages [ 7649.647475] Lustre: 242405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7651.586495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7651.592237] Lustre: Skipped 1 previous similar message [ 7651.597338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7651.638160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7652.146671] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7656.431327] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7658.135584] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7658.235543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7659.922709] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7660.011540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7662.050402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7662.050828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7662.056432] LustreError: Skipped 1 previous similar message [ 7662.057963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7662.064672] Lustre: Skipped 4 previous similar messages [ 7662.073740] Lustre: Skipped 5 previous similar messages [ 7672.290492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7672.293226] Lustre: Skipped 3 previous similar messages [ 7675.360163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7675.425467] Lustre: server umount lustre-OST0000 complete [ 7677.411111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.416459] Lustre: Skipped 1 previous similar message [ 7691.232110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7691.300545] Lustre: server umount lustre-MDT0000 complete [ 7692.681793] LustreError: 240473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789624131 with bad export cookie 16027819962779136585 [ 7692.685844] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7692.686542] LustreError: 240473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7692.802473] Lustre: server umount lustre-MDT0001 complete [ 7695.826953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7695.998316] LustreError: 244618:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.036146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7696.039483] Lustre: Skipped 2 previous similar messages [ 7697.529500] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7700.683124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7702.349685] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7705.092522] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7707.138582] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7709.611815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7711.758524] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 7714.344666] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7859.171586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7859.172501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7859.172788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7859.172792] Lustre: Skipped 6 previous similar messages [ 7859.176037] Lustre: Skipped 1 previous similar message [ 7863.378840] Lustre: server umount lustre-OST0000 complete [ 7879.136158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7879.204340] Lustre: server umount lustre-MDT0000 complete [ 7880.729994] LustreError: 247237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789624319 with bad export cookie 16027819962779137467 [ 7880.733233] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7880.735352] LustreError: 247237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7880.855976] Lustre: server umount lustre-MDT0001 complete [ 7979.244990] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 7980.285738] Key type lgssc unregistered [ 7980.424512] LNet: 248223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7980.427346] LNetError: 248223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7980.441585] LNet: Removed LNI 192.168.206.145@tcp [ 7980.790120] Key type .llcrypt unregistered [ 7980.791132] Key type ._llcrypt unregistered [ 7989.059536] Key type ._llcrypt registered [ 7989.060613] Key type .llcrypt registered [ 7989.099327] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 7995.117222] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 7995.498552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7995.532090] alg: No test for adler32 (adler32-zlib) [ 7996.384386] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 7996.469846] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 7998.048186] Key type lgssc registered [ 7998.448920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8013.554450] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8016.955913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8016.962921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8018.041619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8018.052506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8018.086988] Lustre: lustre-MDT0000: new disk, initializing [ 8018.107230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8018.112887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8019.478678] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8023.604986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8023.643474] Lustre: 252258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8023.665905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8023.667695] Lustre: Skipped 1 previous similar message [ 8023.701189] Lustre: lustre-MDT0001: new disk, initializing [ 8023.735245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8023.748528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8023.752073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8025.078846] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8027.355815] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8029.610796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8029.715018] Lustre: lustre-OST0000: new disk, initializing [ 8029.717506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8029.721534] Lustre: 253890:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8029.754173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8030.456772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8030.461335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8030.499948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8031.628770] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8035.775838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8035.849820] Lustre: lustre-OST0001: new disk, initializing [ 8035.852082] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8035.854432] Lustre: 254763:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8035.892473] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8036.309369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8036.312498] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8036.342302] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8037.830668] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8043.287592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8049.055238] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8054.753051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8054.757102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8054.763068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.801496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.802181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.808990] Lustre: Skipped 2 previous similar messages [ 8058.215664] Lustre: server umount lustre-MDT0000 complete [ 8059.461951] LustreError: 254762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789624498 with bad export cookie 5073502476401892506 [ 8059.465044] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8059.466489] LustreError: 254762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8059.587623] Lustre: server umount lustre-MDT0001 complete [ 8071.113360] Lustre: server umount lustre-OST0000 complete [ 8081.632078] Lustre: server umount lustre-OST0001 complete [ 8086.167988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8089.269337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8092.118486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8095.314102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8101.860602] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8105.982413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.991493] Lustre: lustre-MDT0000: reset Object Index mappings [ 8106.165255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8107.700736] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8110.840207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8110.848655] Lustre: lustre-MDT0001: reset Object Index mappings [ 8112.504256] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8113.513736] Lustre: 260902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8115.887069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8116.042910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8116.045341] Lustre: Skipped 1 previous similar message [ 8117.094029] LustreError: 261254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.096780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8117.100092] LustreError: 261254:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8118.356582] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8121.188136] LustreError: 261255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8121.198941] LustreError: 261255:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8121.461333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8123.678083] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8126.852617] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8126.953548] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8128.071897] Lustre: 262768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8131.553228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8131.556419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.561179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8131.563850] Lustre: Skipped 3 previous similar messages [ 8135.327810] Lustre: server umount lustre-MDT0000 complete [ 8136.608417] LustreError: 260503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789624575 with bad export cookie 5073502476401899282 [ 8136.612203] LustreError: 260503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8136.613074] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8136.731242] Lustre: server umount lustre-MDT0001 complete [ 8148.296316] Lustre: server umount lustre-OST0000 complete [ 8159.838390] Lustre: server umount lustre-OST0001 complete [ 8166.623437] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:56:44 (1789624604) [ 8172.591079] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 8175.865617] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8188.041551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8188.122755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8188.132639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8188.166336] Lustre: lustre-MDT0000: new disk, initializing [ 8188.186461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8188.188890] Lustre: Skipped 1 previous similar message [ 8188.194742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8189.565253] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8193.758971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8193.817990] Lustre: 267011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8193.915421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8195.271187] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8199.159879] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8201.126079] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8203.509260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8203.621519] Lustre: lustre-OST0000: new disk, initializing [ 8203.622807] Lustre: Skipped 1 previous similar message [ 8203.624193] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8203.625816] Lustre: Skipped 2 previous similar messages [ 8203.627564] Lustre: 268981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8204.729522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8204.733028] Lustre: Skipped 1 previous similar message [ 8204.734857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8204.768535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8205.743687] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8209.674389] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8211.230598] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8211.319236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8212.874115] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8212.960241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8215.012846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8215.013390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8215.014892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8215.014899] Lustre: Skipped 4 previous similar messages [ 8215.019261] Lustre: Skipped 4 previous similar messages [ 8215.021527] LustreError: Skipped 1 previous similar message [ 8219.808189] Lustre: server umount lustre-OST0000 complete [ 8224.736942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8224.740325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8224.745658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8224.748468] Lustre: Skipped 1 previous similar message [ 8227.355493] Lustre: server umount lustre-MDT0000 complete [ 8228.725449] LustreError: 268980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789624667 with bad export cookie 5073502476401901347 [ 8228.729293] LustreError: 268980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8228.730810] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8228.841220] Lustre: server umount lustre-MDT0001 complete [ 8235.389502] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8239.037076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8239.172807] LustreError: 271908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8239.186952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8239.189562] Lustre: Skipped 2 previous similar messages [ 8240.544123] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8243.529572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8245.107975] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8246.150485] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8248.446744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8250.449995] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8253.309047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8253.376821] Lustre: lustre-OST0001: new disk, initializing [ 8253.379265] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8253.382662] Lustre: 274214:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8254.653498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8254.656731] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8254.689514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8255.277228] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8259.741422] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8260.952562] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8262.555648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8262.583494] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8265.186421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8265.189442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8265.190194] Lustre: Skipped 4 previous similar messages [ 8265.192162] Lustre: Skipped 4 previous similar messages [ 8271.253960] Lustre: server umount lustre-MDT0000 complete [ 8272.494681] LustreError: 271935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789624711 with bad export cookie 5073502476401902236 [ 8272.499728] LustreError: 271935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8272.499915] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8272.617521] Lustre: server umount lustre-MDT0001 complete [ 8283.593874] Lustre: server umount lustre-OST0000 complete [ 8295.137349] Lustre: server umount lustre-OST0001 complete [ 8300.745340] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8304.390656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8304.530896] LustreError: 277552:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8304.562122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8304.565758] Lustre: Skipped 3 previous similar messages [ 8306.091748] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8309.043059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8310.575172] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8311.541067] Lustre: 278692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8313.823235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8314.981088] LustreError: 279047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8314.986452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8316.068956] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8318.925227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8320.055783] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8321.018971] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8324.157030] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8325.433366] Lustre: 280569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8327.197695] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8327.206561] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8331.554845] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:59:29 (1789624769) [ 8345.056750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8345.059523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8345.065402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8345.067268] Lustre: Skipped 5 previous similar messages [ 8349.211205] Lustre: server umount lustre-MDT0000 complete [ 8350.421209] LustreError: 277533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789624788 with bad export cookie 5073502476401908613 [ 8350.424640] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8350.426718] LustreError: 277533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8350.689173] LustreError: 277549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8350.694501] LustreError: 277549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8356.589974] Lustre: server umount lustre-MDT0001 complete [ 8372.192108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8372.225925] Lustre: server umount lustre-OST0000 complete [ 8379.546861] Lustre: server umount lustre-OST0001 complete [ 8381.683677] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 8385.285175] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8401.023974] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8404.294639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.367571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8404.380191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8404.416321] Lustre: lustre-MDT0000: new disk, initializing [ 8404.440962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8405.693224] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8409.873987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8409.921628] Lustre: 285087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8410.025459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8411.515367] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8416.183220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8416.291198] Lustre: 286721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8418.150552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8418.441485] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8422.880393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8425.009784] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8425.011352] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8429.640115] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8435.386628] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 02:01:13 (1789624873) [ 8440.291261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8440.293145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8440.296768] Lustre: Skipped 8 previous similar messages [ 8440.298875] Lustre: Skipped 9 previous similar messages [ 8442.212996] Lustre: server umount lustre-MDT0000 complete [ 8443.538095] LustreError: 285883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789624882 with bad export cookie 5073502476401916572 [ 8443.541359] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8443.542715] LustreError: 285883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8443.657364] Lustre: server umount lustre-MDT0001 complete [ 8454.600853] Lustre: server umount lustre-OST0000 complete [ 8466.215659] Lustre: server umount lustre-OST0001 complete [ 8469.314648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.511914] LustreError: 289842:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8469.518442] LustreError: 289842:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8469.548109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8469.551718] Lustre: Skipped 7 previous similar messages [ 8471.018773] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8474.080805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8475.752454] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8478.437214] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8480.324820] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8482.633881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8484.674110] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8487.135679] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8502.920812] LustreError: 291672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8502.925668] LustreError: 291672:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 8503.703839] loop1: detected capacity change from 0 to 204800000 [ 8503.790649] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8503.796143] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8503.800093] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8505.461986] loop1: detected capacity change from 0 to 204800000 [ 8507.374627] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.405485] LustreError: 289831:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs19609: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 8507.410949] LustreError: 289831:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs19609 [ 8507.413102] LustreError: 289831:0:(mgs_handler.c:517:mgs_target_reg()) fs19609-MDT0001: Error checking target index: rc = -13 [ 8507.417800] LustreError: fs19609-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8507.420928] LustreError: 293579:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 8507.423552] LustreError: 293579:0:(tgt_mount.c:2128:server_put_super()) no obd fs19609-MDT0001 [ 8507.425991] LustreError: 293579:0:(tgt_mount.c:116:server_deregister_mount()) fs19609-MDT0001 not registered [ 8507.437156] Lustre: server umount fs19609-MDT0001 complete [ 8507.438617] LustreError: 293579:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8510.427195] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8510.459550] Lustre: Setting parameter fs19609-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs19609-MDT0001 [ 8510.463506] Lustre: Skipped 2 previous similar messages [ 8510.479944] Lustre: srv-fs19609-MDT0001: No data found on store. Initialize space. [ 8510.481934] Lustre: Skipped 4 previous similar messages [ 8510.519719] Lustre: fs19609-MDT0001: new disk, initializing [ 8510.521919] Lustre: Skipped 3 previous similar messages [ 8510.559450] Lustre: srv-fs19609-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8510.562488] Lustre: 294045:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19609-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8510.566782] LustreError: 294045:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19609-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8512.028333] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8515.124930] Lustre: server umount fs19609-MDT0001 complete [ 8519.893836] loop1: detected capacity change from 0 to 204800000 [ 8521.813490] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8521.925317] Lustre: 295610:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19609-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8521.929470] LustreError: 295610:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19609-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8523.318828] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8524.962887] Lustre: server umount fs19609-MDT0001 complete [ 8528.526137] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.45@tcp (stopping) [ 8528.528279] Lustre: Skipped 1 previous similar message [ 8530.400949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8530.403155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8534.362455] Lustre: server umount lustre-MDT0000 complete [ 8537.072885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8537.107650] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8538.513481] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8541.036504] loop1: detected capacity change from 0 to 204800000 [ 8542.694018] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8542.828149] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.947296] Lustre: 298052:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs19609-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8542.950414] LustreError: 298052:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs19609-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8544.232735] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8545.772219] Lustre: server umount fs19609-MDT0001 complete [ 8552.932038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8558.224928] Lustre: server umount lustre-OST0000 complete [ 8565.591335] Lustre: server umount lustre-MDT0000 complete [ 8566.798195] LustreError: 294589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789625005 with bad export cookie 5073502476401919246 [ 8566.801877] LustreError: 294589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8566.907544] Lustre: server umount lustre-MDT0001 complete [ 8570.501961] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 8571.516873] Key type lgssc unregistered [ 8571.641474] LNet: 300390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8571.644165] LNetError: 300390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8571.650279] LNet: Removed LNI 192.168.206.145@tcp [ 8571.974108] Key type .llcrypt unregistered [ 8571.975156] Key type ._llcrypt unregistered [ 8579.619309] Key type ._llcrypt registered [ 8579.620627] Key type .llcrypt registered [ 8579.659121] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8580.079885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8580.091031] alg: No test for adler32 (adler32-zlib) [ 8580.946055] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 8581.036500] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 8582.624180] Key type lgssc registered [ 8582.986295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8585.661395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8585.668035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8586.817814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8588.196499] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8591.026913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8591.175735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8592.542620] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8594.930749] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8596.698507] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8598.893093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8599.018253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.792888] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8601.063485] LustreError: 303860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8601.069913] LustreError: 303860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8603.066720] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8606.179434] LustreError: 303462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8611.302509] LustreError: 303464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8613.349451] LustreError: 303462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8613.355712] LustreError: 303462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8618.467401] LustreError: 303464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8618.473261] LustreError: 303464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8619.259516] loop1: detected capacity change from 0 to 204800000 [ 8619.312636] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8619.316940] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8619.320761] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8620.961075] loop1: detected capacity change from 0 to 204800000 [ 8622.723515] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8622.773266] Lustre: fs29455-OST0000: new disk, initializing [ 8622.774925] Lustre: srv-fs29455-OST0000: No data found on store. Initialize space. [ 8622.777501] Lustre: 305386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8622.797566] Lustre: fs29455-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8624.622833] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8627.977944] Lustre: server umount fs29455-OST0000 complete [ 8628.707906] LustreError: 303464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8628.712416] LustreError: 303464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8631.913111] loop1: detected capacity change from 0 to 204800000 [ 8633.760925] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8633.830708] Lustre: fs29455-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8635.734617] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8638.661798] loop2: detected capacity change from 0 to 204800000 [ 8638.731585] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8638.735804] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8638.739657] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8640.638478] loop2: detected capacity change from 0 to 204800000 [ 8642.487307] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8642.517151] LustreError: fs29455-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 8642.520715] LustreError: fs29455-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8642.523342] LustreError: 308422:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 8642.525771] LustreError: 308422:0:(tgt_mount.c:2128:server_put_super()) no obd fs29455-OST0001 [ 8642.527658] LustreError: 308422:0:(tgt_mount.c:116:server_deregister_mount()) fs29455-OST0001 not registered [ 8642.540470] Lustre: server umount fs29455-OST0001 complete [ 8642.541819] LustreError: 308422:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8645.824904] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8645.869107] Lustre: fs29455-OST0001: new disk, initializing [ 8645.870699] Lustre: srv-fs29455-OST0001: No data found on store. Initialize space. [ 8645.873123] Lustre: 308975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8645.894565] Lustre: fs29455-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8647.766807] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8649.187126] LustreError: 303860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8649.191235] LustreError: 303860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 8651.186363] Lustre: server umount fs29455-OST0000 complete [ 8654.253834] Lustre: server umount fs29455-OST0001 complete [ 8658.161594] loop1: detected capacity change from 0 to 204800000 [ 8660.062095] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8662.094276] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8664.274973] loop2: detected capacity change from 0 to 204800000 [ 8666.073611] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8666.127385] Lustre: fs29455-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8666.130436] Lustre: Skipped 1 previous similar message [ 8667.938846] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8670.764373] Lustre: server umount fs29455-OST0000 complete [ 8673.749405] Lustre: server umount fs29455-OST0001 complete [ 8677.856840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8677.860315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8677.865155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8679.906114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8679.908724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8679.909357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8683.483372] Lustre: server umount lustre-OST0000 complete [ 8685.025614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8685.026600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8685.030747] Lustre: Skipped 1 previous similar message [ 8685.032919] Lustre: Skipped 1 previous similar message [ 8690.145542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8690.147341] Lustre: Skipped 2 previous similar messages [ 8695.265021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8695.267085] Lustre: Skipped 1 previous similar message [ 8699.360134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8699.417248] Lustre: server umount lustre-MDT0000 complete [ 8700.384992] LustreError: 301635:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8700.389366] LustreError: 301635:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 8700.556182] LustreError: 301616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789625139 with bad export cookie 11246917896040858379 [ 8700.560077] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8700.560116] LustreError: 301616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8700.668481] Lustre: server umount lustre-MDT0001 complete [ 8704.416457] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 8705.412143] Key type lgssc unregistered [ 8705.542349] LNet: 314883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8705.544807] LNetError: 314883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8705.554373] LNet: Removed LNI 192.168.206.145@tcp [ 8705.861195] Key type .llcrypt unregistered [ 8705.862409] Key type ._llcrypt unregistered [ 8715.963159] Key type ._llcrypt registered [ 8715.964203] Key type .llcrypt registered [ 8716.004621] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 8722.457348] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8722.866311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8722.880022] alg: No test for adler32 (adler32-zlib) [ 8723.731330] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 8723.823364] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 8725.400179] Key type lgssc registered [ 8725.769497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8737.731051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8737.736340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.820583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8738.833077] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8738.868342] Lustre: lustre-MDT0000: new disk, initializing [ 8738.889956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8738.897245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8740.365166] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8744.560690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8744.607865] Lustre: 318709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8744.632479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8744.635063] Lustre: Skipped 1 previous similar message [ 8744.678065] Lustre: lustre-MDT0001: new disk, initializing [ 8744.714129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8744.728140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8744.731864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8746.089831] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8749.776504] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8751.521226] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8753.723224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8753.842374] Lustre: lustre-OST0000: new disk, initializing [ 8753.844106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8753.847659] Lustre: 320673:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8753.876763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8755.450976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8755.455680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8755.500193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8755.859975] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8759.649833] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8760.334725] Lustre: Mounted lustre-client - version 2.17.58_39_ga3f6967 [ 8760.678878] Lustre: Unmounted lustre-client [ 8765.412183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.414045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8765.416593] Lustre: Skipped 1 previous similar message [ 8769.681341] Lustre: server umount lustre-OST0000 complete [ 8775.650548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8775.650870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8775.651593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8775.651598] Lustre: Skipped 1 previous similar message [ 8775.663492] Lustre: Skipped 1 previous similar message [ 8780.768892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8780.770721] Lustre: Skipped 2 previous similar messages [ 8785.376100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8785.438394] Lustre: server umount lustre-MDT0000 complete [ 8786.573267] LustreError: 318700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789625225 with bad export cookie 7114069123202071691 [ 8786.579108] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8786.688483] Lustre: server umount lustre-MDT0001 complete [ 8790.070397] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 8791.103134] Key type lgssc unregistered [ 8816.864428] LNet: 322833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8816.868286] LNetError: 322833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8889.574332] LNet: Removed LNI 192.168.206.145@tcp [ 8889.875633] Key type .llcrypt unregistered [ 8889.877121] Key type ._llcrypt unregistered [ 8895.727161] Key type ._llcrypt registered [ 8895.728496] Key type .llcrypt registered [ 8895.763104] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 8902.367766] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8902.710845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8902.720828] alg: No test for adler32 (adler32-zlib) [ 8903.566039] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 8903.656033] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 8905.232171] Key type lgssc registered [ 8905.577920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8915.904054] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:09:14 (1789625354) [ 8918.257407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8918.263239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8919.347715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8919.358616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8919.395452] Lustre: lustre-MDT0000: new disk, initializing [ 8919.420809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8919.428551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8920.841871] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8925.310593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8925.354384] Lustre: 326036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8925.379253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8925.381922] Lustre: Skipped 1 previous similar message [ 8925.402042] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8925.423545] Lustre: lustre-MDT0001: new disk, initializing [ 8925.457965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8925.473951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8925.477887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8926.953573] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8930.842178] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8932.672421] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8934.833639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8934.942901] Lustre: lustre-OST0000: new disk, initializing [ 8934.944578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8934.946889] Lustre: 328002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8934.977172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8936.977122] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8939.515836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8939.520318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8939.553583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8940.765083] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8949.730066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8949.730633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8949.732710] LustreError: Skipped 1 previous similar message [ 8949.738252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8949.741555] Lustre: Skipped 1 previous similar message [ 8952.725738] Lustre: server umount lustre-OST0000 complete [ 8954.849601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8954.850155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8954.853820] Lustre: Skipped 2 previous similar messages [ 8959.969527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8959.971972] Lustre: Skipped 2 previous similar messages [ 8960.149203] Lustre: server umount lustre-MDT0000 complete [ 8961.329415] LustreError: 326028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789625399 with bad export cookie 16232095489835391493 [ 8961.332267] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8961.333280] LustreError: 326028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8961.451748] Lustre: server umount lustre-MDT0001 complete [ 8964.936968] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 8965.941996] Key type lgssc unregistered [ 8966.068526] LNet: 329907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8966.071155] LNetError: 329907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8966.080318] LNet: Removed LNI 192.168.206.145@tcp [ 8966.371045] Key type .llcrypt unregistered [ 8966.372479] Key type ._llcrypt unregistered [ 8974.100072] Key type ._llcrypt registered [ 8974.101204] Key type .llcrypt registered [ 8974.138265] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 8974.518961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8974.538471] alg: No test for adler32 (adler32-zlib) [ 8975.386872] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 8975.479717] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 8977.056181] Key type lgssc registered [ 8977.423300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8980.003125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8980.009859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8981.143678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8982.386299] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8985.046417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8985.155674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8986.427424] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8988.791820] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8990.544899] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8992.772566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8992.888725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8994.769820] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 8997.215526] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9011.032609] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9011.548401] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9012.877156] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.45@tcp (stopping) [ 9015.777035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9015.779429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9015.784201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9016.801195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9016.802773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9016.803550] Lustre: Skipped 1 previous similar message [ 9016.808543] Lustre: Skipped 1 previous similar message [ 9018.975646] Lustre: server umount lustre-MDT0000 complete [ 9020.229257] LustreError: 331134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789625458 with bad export cookie 6015448062946530212 [ 9020.234199] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9020.234332] LustreError: 331134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9021.921024] LustreError: 332700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9021.921467] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9021.926683] LustreError: 332700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9021.933394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9021.935529] Lustre: Skipped 2 previous similar messages [ 9023.111631] LustreError: 332992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9025.952943] LustreError: 331149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9025.953271] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9025.958948] Lustre: Skipped 1 previous similar message [ 9026.409370] Lustre: server umount lustre-MDT0001 complete [ 9033.683426] Lustre: server umount lustre-OST0000 complete [ 9036.415345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9037.736954] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9038.627518] Lustre: 335352:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.145@tcp [ 9039.332180] Lustre: server umount MGS complete [ 9042.040899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9042.180697] LustreError: 335921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9042.201645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.424452] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9046.068041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.384026] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9050.075543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9050.189702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9050.192319] Lustre: Skipped 1 previous similar message [ 9052.144664] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9054.556838] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9055.050813] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9056.790416] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9057.271171] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9061.859435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9061.859598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9061.860506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9061.863246] Lustre: Skipped 1 previous similar message [ 9062.035117] Lustre: server umount lustre-OST0000 complete [ 9065.360908] Lustre: server umount lustre-MDT0000 complete [ 9066.569340] LustreError: 335902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789625505 with bad export cookie 6015448062946531038 [ 9066.573403] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9066.575193] LustreError: 335902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9066.680066] Lustre: server umount lustre-MDT0001 complete [ 9070.016695] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9071.048837] Key type lgssc unregistered [ 9071.167406] LNet: 339349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9071.169933] LNetError: 339349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9071.177388] LNet: Removed LNI 192.168.206.145@tcp [ 9071.503504] Key type .llcrypt unregistered [ 9071.504809] Key type ._llcrypt unregistered [ 9077.216472] Key type ._llcrypt registered [ 9077.217673] Key type .llcrypt registered [ 9077.258668] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 9083.319018] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 9083.734877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9083.742427] alg: No test for adler32 (adler32-zlib) [ 9084.599989] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 9084.688466] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 9086.264202] Key type lgssc registered [ 9086.593966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9098.059072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9098.066142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9099.155658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9099.167032] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9099.200496] Lustre: lustre-MDT0000: new disk, initializing [ 9099.219863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.227478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9100.595174] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9104.891613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9104.936456] Lustre: 342478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9104.958967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9104.961110] Lustre: Skipped 1 previous similar message [ 9104.997943] Lustre: lustre-MDT0001: new disk, initializing [ 9105.034554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9105.047903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9105.053582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9106.448438] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9110.231858] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9112.110777] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9114.510537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9114.620578] Lustre: lustre-OST0000: new disk, initializing [ 9114.622313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9114.625077] Lustre: 344444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9114.658819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9114.939121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9114.943583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9114.988782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9116.676138] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9120.347928] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9121.716841] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9121.787364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9123.104181] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9123.176040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9125.345453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9125.346970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9125.348153] LustreError: Skipped 1 previous similar message [ 9125.352831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9125.356631] Lustre: Skipped 1 previous similar message [ 9130.466843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9130.469694] Lustre: Skipped 1 previous similar message [ 9135.587030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9135.589465] Lustre: Skipped 1 previous similar message [ 9138.144091] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9138.200961] Lustre: server umount lustre-OST0000 complete [ 9140.705592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9140.706312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9140.710083] Lustre: Skipped 2 previous similar messages [ 9140.713886] Lustre: Skipped 1 previous similar message [ 9145.627041] Lustre: server umount lustre-MDT0000 complete [ 9146.828513] LustreError: 343272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789625585 with bad export cookie 13875175767175574931 [ 9146.831788] LustreError: 343272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9146.831916] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9146.943388] Lustre: server umount lustre-MDT0001 complete [ 9149.734941] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:13:08 (1789625588) [ 9155.147990] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 9158.463875] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 9170.357253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9170.445371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9170.457262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9170.494820] Lustre: lustre-MDT0000: new disk, initializing [ 9170.518954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9170.524760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9171.946540] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9176.079555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9176.121909] Lustre: 349193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9176.147665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9176.149946] Lustre: Skipped 1 previous similar message [ 9176.185277] Lustre: lustre-MDT0001: new disk, initializing [ 9176.219620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9176.231807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9176.234684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9177.608617] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9181.217957] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9183.043402] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9185.183792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9185.288792] Lustre: lustre-OST0000: new disk, initializing [ 9185.290891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9185.294220] Lustre: 351161:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9185.317558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9187.223697] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9187.263670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9187.267156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9187.304347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9190.859995] Lustre: DEBUG MARKER: oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9202.658996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9202.659370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9202.660178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9202.663738] Lustre: Skipped 1 previous similar message [ 9202.667105] LustreError: Skipped 1 previous similar message [ 9206.090729] Lustre: server umount lustre-OST0000 complete [ 9207.776938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.384766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9212.387724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.392553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9212.394092] Lustre: Skipped 2 previous similar messages [ 9213.461083] Lustre: server umount lustre-MDT0000 complete [ 9214.630338] LustreError: 349186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789625653 with bad export cookie 13875175767175575799 [ 9214.633451] LustreError: 349186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9214.633726] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9214.752254] Lustre: server umount lustre-MDT0001 complete [ 9218.373083] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9219.364579] Key type lgssc unregistered [ 9219.480332] LNet: 353308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9219.482429] LNetError: 353308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9219.490262] LNet: Removed LNI 192.168.206.145@tcp [ 9219.778971] Key type .llcrypt unregistered [ 9219.780227] Key type ._llcrypt unregistered [ 9250.601615] Key type ._llcrypt registered [ 9250.602444] Key type .llcrypt registered [ 9250.635492] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9250.734449] Key type .llcrypt unregistered [ 9250.735715] Key type ._llcrypt unregistered [ 9277.028505] Key type ._llcrypt registered [ 9277.029466] Key type .llcrypt registered [ 9277.067397] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9277.182446] Key type .llcrypt unregistered [ 9277.183983] Key type ._llcrypt unregistered [ 9288.241867] Key type ._llcrypt registered [ 9288.242869] Key type .llcrypt registered [ 9288.281582] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9288.376224] Key type .llcrypt unregistered [ 9288.377271] Key type ._llcrypt unregistered [ 9299.620763] Key type ._llcrypt registered [ 9299.622559] Key type .llcrypt registered [ 9299.660709] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9299.763249] Key type .llcrypt unregistered [ 9299.764876] Key type ._llcrypt unregistered [ 9326.486216] Key type ._llcrypt registered [ 9326.487268] Key type .llcrypt registered [ 9326.520255] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9326.610388] Key type .llcrypt unregistered [ 9326.611499] Key type ._llcrypt unregistered [ 9337.513869] Key type ._llcrypt registered [ 9337.515246] Key type .llcrypt registered [ 9337.552220] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9337.653198] Key type .llcrypt unregistered [ 9337.654538] Key type ._llcrypt unregistered [ 9348.664445] Key type ._llcrypt registered [ 9348.665870] Key type .llcrypt registered [ 9348.703466] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9348.806387] Key type .llcrypt unregistered [ 9348.807890] Key type ._llcrypt unregistered [ 9359.274769] Key type ._llcrypt registered [ 9359.276358] Key type .llcrypt registered [ 9359.316955] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9359.414403] Key type .llcrypt unregistered [ 9359.415680] Key type ._llcrypt unregistered [ 9370.110921] Key type ._llcrypt registered [ 9370.111903] Key type .llcrypt registered [ 9370.145794] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9370.241345] Key type .llcrypt unregistered [ 9370.242185] Key type ._llcrypt unregistered [ 9380.954770] Key type ._llcrypt registered [ 9380.956120] Key type .llcrypt registered [ 9380.993816] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9381.097367] Key type .llcrypt unregistered [ 9381.098565] Key type ._llcrypt unregistered [ 9401.855269] Key type ._llcrypt registered [ 9401.856497] Key type .llcrypt registered [ 9401.896630] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9402.003476] Key type .llcrypt unregistered [ 9402.004755] Key type ._llcrypt unregistered [ 9412.584767] Key type ._llcrypt registered [ 9412.586211] Key type .llcrypt registered [ 9412.621493] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9412.716377] Key type .llcrypt unregistered [ 9412.717364] Key type ._llcrypt unregistered [ 9423.373512] Key type ._llcrypt registered [ 9423.374557] Key type .llcrypt registered [ 9423.410478] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9423.504199] Key type .llcrypt unregistered [ 9423.505226] Key type ._llcrypt unregistered [ 9440.633852] Key type ._llcrypt registered [ 9440.634944] Key type .llcrypt registered [ 9440.671930] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 9447.123739] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 9447.607966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9447.616860] alg: No test for adler32 (adler32-zlib) [ 9448.472119] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 9448.560058] LNet: Added LNI 192.168.206.145@tcp [8/256/0/180] [ 9450.144190] Key type lgssc registered [ 9450.504633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9461.870679] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9461.876904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.949624] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 9462.959594] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 9462.990196] Lustre: tst250fs-MDT0000: new disk, initializing [ 9463.009080] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9463.015335] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9464.260176] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9468.123225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9468.161797] Lustre: 373160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9468.180330] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 9468.182111] Lustre: Skipped 1 previous similar message [ 9468.212153] Lustre: tst250fs-MDT0001: new disk, initializing [ 9468.239235] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9468.249247] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9468.251933] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9469.493204] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9471.659355] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9473.662655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9473.747797] Lustre: tst250fs-OST0000: new disk, initializing [ 9473.749291] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 9473.751645] Lustre: 374789:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9473.781826] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9475.627112] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9478.648793] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9478.651701] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9478.690555] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9479.603512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9479.662848] Lustre: tst250fs-OST0001: new disk, initializing [ 9479.664958] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 9479.667231] Lustre: 375662:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9479.696098] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9481.447159] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_default_debug -1 all [ 9483.972633] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 9488.373446] Lustre: 373162:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.145@tcp: IR log tst250fs-mdtir failed, not fatal: rc = -2 [ 9495.011395] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9495.012576] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9495.015778] Lustre: Skipped 3 previous similar messages [ 9495.017604] Lustre: Skipped 2 previous similar messages [ 9500.129762] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9500.133048] Lustre: Skipped 4 previous similar messages [ 9500.318457] Lustre: server umount tst250fs-MDT0000 complete [ 9501.572234] LustreError: 373955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789625940 with bad export cookie 12210934315060533442 [ 9501.576274] LustreError: 373955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9501.577101] LustreError: MGC192.168.206.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9501.701984] Lustre: server umount tst250fs-MDT0001 complete [ 9513.480965] Lustre: server umount tst250fs-OST0000 complete [ 9525.084354] Lustre: server umount tst250fs-OST0001 complete [ 9531.115759] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 9533.530910] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 9542.886914] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:19:41 (1789625981) [ 9543.341995] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 9548.244220] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing set_hostid [ 9551.200720] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing load_modules_local [ 9560.655095] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9294 sec ========== 02:19:59 (1789625999) [ 9561.215505] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:19:59 (1789625999) === [ 9562.311829] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:20:00 (1789626000) === [ 9574.436173] Lustre: DEBUG MARKER: oleg645-server.virtnet: executing unload_modules_local [ 9575.358876] Key type lgssc unregistered [ 9575.468335] LNet: 384761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9575.470487] LNetError: 384761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9575.478285] LNet: Removed LNI 192.168.206.145@tcp [ 9575.763136] Key type .llcrypt unregistered [ 9575.764387] Key type ._llcrypt unregistered