[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 846685064 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003259] x2apic enabled [ 0.005007] Switched APIC routing to physical x2apic. [ 0.006025] kvm-guest: setup PV IPIs [ 0.010412] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.011000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012021] pid_max: default: 32768 minimum: 301 [ 0.014113] LSM: Security Framework initializing [ 0.015083] Yama: becoming mindful. [ 0.016066] SELinux: Initializing. [ 0.017130] *** VALIDATE selinux *** [ 0.027552] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.035052] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.037251] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.038201] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.039140] *** VALIDATE tmpfs *** [ 0.041723] *** VALIDATE proc *** [ 0.045045] *** VALIDATE cgroup *** [ 0.046017] *** VALIDATE cgroup2 *** [ 0.048244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.049204] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.050019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.051064] Spectre V2 : User space: Vulnerable [ 0.052014] Speculative Store Bypass: Vulnerable [ 0.056188] debug: unmapping init [mem 0xffffffff9f259000-0xffffffff9f260fff] [ 0.060380] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.062685] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.063032] ... version: 2 [ 0.064024] ... bit width: 48 [ 0.065025] ... generic registers: 4 [ 0.066019] ... value mask: 0000ffffffffffff [ 0.067026] ... max period: 00007fffffffffff [ 0.068028] ... fixed-purpose events: 3 [ 0.069021] ... event mask: 000000070000000f [ 0.070379] rcu: Hierarchical SRCU implementation. [ 0.073723] smp: Bringing up secondary CPUs ... [ 0.075158] x86: Booting SMP configuration: [ 0.076040] .... node #0, CPUs: #1 #2 #3 [ 0.088022] smp: Brought up 1 node, 4 CPUs [ 0.090021] smpboot: Max logical packages: 1 [ 0.091025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.161759] node 0 deferred pages initialised in 68ms [ 0.171346] devtmpfs: initialized [ 0.174000] x86/mm: Memory block size: 128MB [ 0.175594] gcov: version magic: 0x41383552 [ 0.183104] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.184142] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.187647] pinctrl core: initialized pinctrl subsystem [ 0.189664] [ 0.191012] ************************************************************* [ 0.194021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198022] ** ** [ 0.203018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.206015] ** ** [ 0.210020] ** This means that this kernel is built to expose internal ** [ 0.214114] ** IOMMU data structures, which may compromise security on ** [ 0.218016] ** your system. ** [ 0.221018] ** ** [ 0.225018] ** If you see this message and you are not debugging the ** [ 0.228018] ** kernel, report this immediately to your vendor! ** [ 0.232020] ** ** [ 0.236021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.237025] ************************************************************* [ 0.241598] NET: Registered protocol family 16 [ 0.243796] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.247137] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.252149] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.257192] cpuidle: using governor menu [ 0.261340] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.268000] PCI: Using configuration type 1 for base access [ 0.271230] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.286310] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.287028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.291975] cryptd: max_cpu_qlen set to 1000 [ 0.293270] ACPI: Added _OSI(Module Device) [ 0.294018] ACPI: Added _OSI(Processor Device) [ 0.295021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.297022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.303890] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.315807] ACPI: Interpreter enabled [ 0.318235] ACPI: PM: (supports S0 S3 S4 S5) [ 0.320016] ACPI: Using IOAPIC for interrupt routing [ 0.323140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.328776] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.340689] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.345056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.349030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.352341] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.361452] acpiphp: Slot [2] registered [ 0.364562] acpiphp: Slot [5] registered [ 0.367599] acpiphp: Slot [6] registered [ 0.370479] acpiphp: Slot [7] registered [ 0.373393] acpiphp: Slot [8] registered [ 0.378180] acpiphp: Slot [9] registered [ 0.380205] acpiphp: Slot [10] registered [ 0.381156] acpiphp: Slot [3] registered [ 0.384165] acpiphp: Slot [4] registered [ 0.387859] acpiphp: Slot [11] registered [ 0.388000] acpiphp: Slot [12] registered [ 0.389163] acpiphp: Slot [13] registered [ 0.397186] acpiphp: Slot [14] registered [ 0.400171] acpiphp: Slot [15] registered [ 0.403401] acpiphp: Slot [16] registered [ 0.407353] acpiphp: Slot [17] registered [ 0.408000] acpiphp: Slot [18] registered [ 0.409260] acpiphp: Slot [19] registered [ 0.411349] acpiphp: Slot [20] registered [ 0.414148] acpiphp: Slot [21] registered [ 0.416186] acpiphp: Slot [22] registered [ 0.419246] acpiphp: Slot [23] registered [ 0.422564] acpiphp: Slot [24] registered [ 0.425134] acpiphp: Slot [25] registered [ 0.428168] acpiphp: Slot [26] registered [ 0.429374] acpiphp: Slot [27] registered [ 0.432232] acpiphp: Slot [28] registered [ 0.433219] acpiphp: Slot [29] registered [ 0.434122] acpiphp: Slot [30] registered [ 0.436170] acpiphp: Slot [31] registered [ 0.438076] PCI host bridge to bus 0000:00 [ 0.439025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.442030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.444025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.446030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.448144] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.451050] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.452919] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.455000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.458149] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.473023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.482000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.484023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.487026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.493026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.500866] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.506707] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.510049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.515000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.519030] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.539021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.551018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.564799] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.578021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.589019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.610026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.629089] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.639020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.647020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.664021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.679583] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.688020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.696131] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.715021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.725012] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.731021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.736017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.759024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.783760] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.796024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.817023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.856022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.881314] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.889023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.899023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.930019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.946852] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.952093] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.955431] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.957720] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.961069] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.973041] iommu: Default domain type: Passthrough [ 0.983010] SCSI subsystem initialized [ 0.986176] ACPI: bus type USB registered [ 0.991161] usbcore: registered new interface driver usbfs [ 0.996120] usbcore: registered new interface driver hub [ 1.001000] usbcore: registered new device driver usb [ 1.002210] pps_core: LinuxPPS API ver. 1 registered [ 1.006014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.009065] PTP clock support registered [ 1.013138] EDAC MC: Ver: 3.0.0 [ 1.017196] PCI: Using ACPI for IRQ routing [ 1.025389] NetLabel: Initializing [ 1.026021] NetLabel: domain hash size = 128 [ 1.027016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.030084] NetLabel: unlabeled traffic allowed by default [ 1.036149] vgaarb: loaded [ 1.041333] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.048016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.063285] clocksource: Switched to clocksource kvm-clock [ 1.438967] VFS: Disk quotas dquot_6.6.0 [ 1.453018] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.471841] *** VALIDATE ramfs *** [ 1.473341] *** VALIDATE hugetlbfs *** [ 1.474741] pnp: PnP ACPI init [ 1.483542] pnp: PnP ACPI: found 6 devices [ 1.535360] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.545445] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.550150] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.557108] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.563673] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.570126] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.577413] NET: Registered protocol family 2 [ 1.585044] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.597613] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.606945] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.619690] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.631383] TCP: Hash tables configured (established 65536 bind 65536) [ 1.641318] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.649980] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.656658] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.666437] NET: Registered protocol family 1 [ 1.677446] RPC: Registered named UNIX socket transport module. [ 1.682042] RPC: Registered udp transport module. [ 1.687041] RPC: Registered tcp transport module. [ 1.690337] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.696638] NET: Registered protocol family 44 [ 1.699672] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.703456] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.709584] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.715740] PCI: CLS 0 bytes, default 64 [ 1.723290] Unpacking initramfs... [ 4.655445] debug: unmapping init [mem 0xffff8a12fcc54000-0xffff8a12fffbffff] [ 4.662895] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.667598] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.673490] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.531345] Initialise system trusted keyrings [ 6.533548] Key type blacklist registered [ 6.536951] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.555809] zbud: loaded [ 6.559934] *** VALIDATE nfs *** [ 6.561954] *** VALIDATE nfs4 *** [ 6.564501] pstore: using deflate compression [ 6.574322] Platform Keyring initialized [ 6.874838] NET: Registered protocol family 38 [ 6.877888] Key type asymmetric registered [ 6.879780] Asymmetric key parser 'x509' registered [ 6.883310] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.890773] io scheduler mq-deadline registered [ 6.896212] io scheduler kyber registered [ 6.900682] io scheduler bfq registered [ 6.905509] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.912750] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.916771] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.920756] ACPI: Power Button [PWRF] [ 6.928448] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.940694] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.962960] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.975508] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.026357] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.061865] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.105582] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.123974] Non-volatile memory driver v1.3 [ 7.126262] Linux agpgart interface v0.103 [ 7.185439] virtio_blk virtio1: [vda] 146656 512-byte logical blocks (75.1 MB/71.6 MiB) [ 7.189104] vda: detected capacity change from 0 to 75087872 [ 7.221193] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.227252] vdb: detected capacity change from 0 to 1073741824 [ 7.267595] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.276380] vdc: detected capacity change from 0 to 2621440000 [ 7.303710] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.307750] vdd: detected capacity change from 0 to 2621440000 [ 7.339118] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.346917] vde: detected capacity change from 0 to 4294967296 [ 7.398279] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.407686] vdf: detected capacity change from 0 to 4294967296 [ 7.422725] libphy: Fixed MDIO Bus: probed [ 7.442917] usbcore: registered new interface driver usbserial_generic [ 7.450782] usbserial: USB Serial support registered for generic [ 7.456287] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.464503] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.472221] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.476679] mousedev: PS/2 mouse device common for all mice [ 7.481351] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.482546] rtc_cmos 00:05: RTC can wake from S4 [ 7.492901] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.495966] rtc_cmos 00:05: registered as rtc0 [ 7.508719] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.515991] intel_pstate: CPU model not supported [ 7.525083] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.531300] hid: raw HID events driver (C) Jiri Kosina [ 7.546216] usbcore: registered new interface driver usbhid [ 7.551055] usbhid: USB HID core driver [ 7.554703] drop_monitor: Initializing network drop monitor service [ 7.559569] Initializing XFRM netlink socket [ 7.563843] NET: Registered protocol family 10 [ 7.570418] Segment Routing with IPv6 [ 7.571902] NET: Registered protocol family 17 [ 7.577237] mpls_gso: MPLS GSO support [ 7.587202] RAS: Correctable Errors collector initialized. [ 7.591698] AVX version of gcm_enc/dec engaged. [ 7.594740] AES CTR mode by8 optimization enabled [ 7.757385] sched_clock: Marking stable (7757362381, 0)->(9810458181, -2053095800) [ 7.762306] registered taskstats version 1 [ 7.765576] Loading compiled-in X.509 certificates [ 7.767948] zswap: loaded using pool lzo/zbud [ 7.814665] Key type big_key registered [ 7.837978] Key type encrypted registered [ 7.840482] ima: No TPM chip found, activating TPM-bypass! [ 7.843608] ima: Allocated hash algorithm: sha1 [ 7.847293] ima: No architecture policies found [ 7.849987] evm: Initialising EVM extended attributes: [ 7.852620] evm: security.selinux [ 7.854581] evm: security.ima [ 7.856552] evm: security.capability [ 7.858595] evm: HMAC attrs: 0x1 [ 7.862625] rtc_cmos 00:05: setting system clock to 2026-09-08 01:17:47 UTC (1788830267) [ 7.872627] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 7.877577] debug: unmapping init [mem 0xffffffff9ef82000-0xffffffff9f258fff] [ 7.888227] Write protecting the kernel read-only data: 28672k [ 7.893526] debug: unmapping init [mem 0xffffffff9d603000-0xffffffff9d7fffff] [ 7.898421] debug: unmapping init [mem 0xffffffff9df14000-0xffffffff9dffffff] [ 7.980567] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.999161] systemd[1]: Detected virtualization kvm. [ 8.002332] systemd[1]: Detected architecture x86-64. [ 8.006558] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.041339] systemd[1]: No hostname configured. [ 8.044577] systemd[1]: Set hostname to . [ 8.050088] random: systemd: uninitialized urandom read (16 bytes read) [ 8.056518] systemd[1]: Initializing machine ID from random generator. [ 8.187118] random: ln: uninitialized urandom read (6 bytes read) [ 8.448722] random: systemd: uninitialized urandom read (16 bytes read) [ 8.455572] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 8.487670] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 8.552731] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.004615] device-mapper: uevent: version 1.0.3 [ 10.007532] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.875863] virtio_net virtio0 ens2: renamed from eth0 [ 12.017311] random: fast init done [ 12.278258] scsi host0: ata_piix [ 12.372707] scsi host1: ata_piix [ 12.383706] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.397128] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.403593] random: crng init done [ 17.422648] random: 7 urandom warning(s) missed due to ratelimiting [ 20.242730] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.720038] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.001227] printk: systemd: 25 output lines suppressed due to ratelimiting [ 27.018938] SELinux: Disabled at runtime. [ 27.178680] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.196412] systemd[1]: Detected virtualization kvm. [ 27.212882] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 29.613822] systemd[1]: initrd-switch-root.service: Succeeded. [ 29.629187] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 29.659761] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 29.677229] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.689709] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.712491] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.735897] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ 30.683286] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 31.827381] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 33.470173] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.500055] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 34.117447] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 34.224954] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit)[ 38.968769] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 40.150729] NFS: Registering the id_resolver key type [ 40.160873] Key type id_resolver registered [ 40.166831] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg402-server login: [ 95.373179] libcfs: loading out-of-tree module taints kernel. [ 95.407892] Key type ._llcrypt registered [ 95.409434] Key type .llcrypt registered [ 95.483383] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 108.934767] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 109.883277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 109.902855] alg: No test for adler32 (adler32-zlib) [ 111.180883] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 111.823993] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 113.479181] Key type lgssc registered [ 114.733387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.646081] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 159.860402] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 169.414732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 169.441519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.622977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 170.657973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 170.733065] Lustre: lustre-MDT0000: new disk, initializing [ 170.795347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.821285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 174.060458] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.613821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.735854] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.774128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 184.783302] Lustre: Skipped 1 previous similar message [ 184.872401] Lustre: lustre-MDT0001: new disk, initializing [ 184.927759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.948886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 184.958368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 188.312050] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.408675] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.588434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.797388] Lustre: lustre-OST0000: new disk, initializing [ 199.800827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.806057] Lustre: 8443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.862284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.308623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.316801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.389726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.790026] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.423754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.518722] Lustre: lustre-OST0001: new disk, initializing [ 216.521564] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.526972] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.568529] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.340107] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.320712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 226.329765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 226.392645] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.010607] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 237.976586] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 243.244468] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing check_logdir /tmp/testlogs/ [ 247.205561] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing yml_node [ 250.316855] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 252.222809] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 254.117441] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 255.400976] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 21:21:54 EDT 2026 [ 268.189512] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 269.415158] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 287.201483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 287.212742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.222574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.714142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.731556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.743617] Lustre: Skipped 2 previous similar messages [ 290.070578] Lustre: server umount lustre-MDT0000 complete [ 294.133845] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788830553 with bad export cookie 13315655499655561510 [ 294.149694] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.149960] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 297.956814] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.958726] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.965656] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 297.988235] Lustre: Skipped 2 previous similar messages [ 297.992028] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.997537] Lustre: Skipped 1 previous similar message [ 300.000620] LustreError: 8229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.000714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.516500] Lustre: server umount lustre-MDT0001 complete [ 304.277523] Lustre: server umount lustre-OST0000 complete [ 308.056172] Lustre: server umount lustre-OST0001 complete [ 314.566662] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 322.083788] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 352.129362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.345086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 352.376320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 352.465622] Lustre: lustre-MDT0000: new disk, initializing [ 352.519292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 352.534991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 356.375502] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 359.843018] hrtimer: interrupt took 5008724 ns [ 365.716792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 365.836747] Lustre: 15063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 365.913863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 365.922712] Lustre: Skipped 1 previous similar message [ 366.003468] Lustre: lustre-MDT0001: new disk, initializing [ 366.097440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 366.129402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 366.136354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 369.511627] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 377.396440] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 382.417708] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 387.967147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 388.204502] Lustre: lustre-OST0000: new disk, initializing [ 388.212311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 388.223222] Lustre: 17029:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 388.305345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.035381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 390.048825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 390.176953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 393.717489] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 402.660977] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 406.695605] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 406.861722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 410.272475] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 410.402049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 415.716746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 415.717554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.719601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 415.719609] Lustre: Skipped 1 previous similar message [ 415.723339] LustreError: Skipped 1 previous similar message [ 415.741024] Lustre: Skipped 2 previous similar messages [ 417.788616] Lustre: server umount lustre-OST0000 complete [ 420.836524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.957400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 425.965460] Lustre: Skipped 4 previous similar messages [ 435.167131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 435.287377] Lustre: server umount lustre-MDT0000 complete [ 436.194382] LustreError: 15933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 436.207796] LustreError: 15933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 438.782491] LustreError: 15055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788830698 with bad export cookie 13315655499655563043 [ 438.790356] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 438.791364] LustreError: 15055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 439.025191] Lustre: server umount lustre-MDT0001 complete [ 443.869649] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:25:03 (1788830703) [ 457.258180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.633219] LustreError: 19904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.725387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.171589] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 462.821679] LustreError: 19905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.278110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.325361] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 475.934895] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 480.044616] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 481.783958] Lustre: Failing over lustre-MDT0000 [ 481.795488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.801813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.806592] Lustre: Skipped 2 previous similar messages [ 481.816631] Lustre: Skipped 3 previous similar messages [ 482.033167] Lustre: server umount lustre-MDT0000 complete [ 486.881149] LustreError: 19905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.903184] LustreError: 19905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 498.686461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.771886] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 502.103603] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 504.292464] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 504.297121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 504.313922] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 509.338747] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 513.505278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.511228] Lustre: Skipped 1 previous similar message [ 518.627602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 518.634048] Lustre: Skipped 3 previous similar messages [ 518.690416] Lustre: server umount lustre-MDT0000 complete [ 521.319792] LustreError: 19884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788830780 with bad export cookie 13315655499655564457 [ 521.322348] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.330221] LustreError: 19884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 521.551985] Lustre: server umount lustre-MDT0001 complete [ 527.396789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.738098] LustreError: 23238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.762593] LustreError: 23238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 527.804298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.809717] Lustre: Skipped 2 previous similar messages [ 530.775306] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 537.099137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.606255] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 545.979821] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 549.945687] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 551.692885] Lustre: Failing over lustre-MDT0000 [ 551.905703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.918099] Lustre: Skipped 1 previous similar message [ 551.941405] Lustre: server umount lustre-MDT0000 complete [ 562.145627] LustreError: 23234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.174263] LustreError: 23234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 571.813315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.172091] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.028940] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 578.063413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 578.067290] Lustre: Skipped 1 previous similar message [ 578.116772] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 578.841656] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 591.665941] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 598.500543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 598.502878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 598.524990] Lustre: Skipped 3 previous similar messages [ 611.295151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 612.378378] Lustre: server umount lustre-MDT0000 complete [ 620.354105] LustreError: 23220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788830879 with bad export cookie 13315655499655565395 [ 620.377780] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 620.391558] LustreError: 23220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 621.110585] Lustre: server umount lustre-MDT0001 complete [ 633.389872] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:28:11 (1788830891) [ 654.554924] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 672.479189] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 711.715973] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 723.182243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.467818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 723.517258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 723.611527] Lustre: lustre-MDT0000: new disk, initializing [ 723.717684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 723.727222] Lustre: Skipped 2 previous similar messages [ 723.745403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 728.718864] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 739.597954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.748349] Lustre: 29642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 740.081604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 744.396168] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 754.942336] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 761.612692] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 768.421762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 768.760789] Lustre: 31610:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 770.440579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 770.486656] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 770.605056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 770.629865] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 775.153338] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 784.679349] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 790.111992] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 811.492753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 811.494181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 811.519296] Lustre: Skipped 2 previous similar messages [ 811.530300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 811.537693] Lustre: Skipped 5 previous similar messages [ 824.287297] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 824.512378] Lustre: server umount lustre-OST0000 complete [ 834.481766] Lustre: server umount lustre-MDT0000 complete [ 837.089255] LustreError: 29653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 837.122297] LustreError: 29653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 839.080716] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831098 with bad export cookie 13315655499655565801 [ 839.088126] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 839.089340] LustreError: 29633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 839.678243] Lustre: server umount lustre-MDT0001 complete [ 851.669617] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 854.681843] Key type lgssc unregistered [ 855.079702] LNet: 33589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.100135] LNetError: 33589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.119554] LNet: Removed LNI 192.168.204.102@tcp [ 856.037284] Key type .llcrypt unregistered [ 856.039335] Key type ._llcrypt unregistered [ 883.719743] Key type ._llcrypt registered [ 883.724295] Key type .llcrypt registered [ 883.867910] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 885.556520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 885.721572] alg: No test for adler32 (adler32-zlib) [ 886.895468] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 887.236538] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 888.978451] Key type lgssc registered [ 890.397863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 899.120109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 899.145104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 900.585085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.760212] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 910.827189] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 910.856125] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 915.207905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.619173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 921.517381] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 930.198633] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 936.598438] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 943.948777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 944.356740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.180868] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 959.983541] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 960.821252] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 975.882038] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:33:54 (1788831234) [ 978.500388] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 980.332251] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 987.905422] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:34:06 (1788831246) [ 997.822974] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:34:16 (1788831256) [ 999.233844] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1009.692782] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1009.702974] Lustre: Skipped 2 previous similar messages [ 1011.666476] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:34:30 (1788831270) [ 1014.036969] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1014.047423] Lustre: Skipped 2 previous similar messages [ 1023.835718] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1023.840988] Lustre: Skipped 4 previous similar messages [ 1039.223792] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:34:57 (1788831297) [ 1042.550662] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1047.025655] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1057.853286] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:35:16 (1788831316) [ 1060.175700] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1060.181962] Lustre: Skipped 6 previous similar messages [ 1074.991304] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:35:33 (1788831333) [ 1094.118440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1094.119206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1094.146534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1094.153821] Lustre: Skipped 1 previous similar message [ 1097.476598] Lustre: server umount lustre-OST0000 complete [ 1104.370904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.374099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1104.393599] Lustre: Skipped 2 previous similar messages [ 1104.416492] Lustre: Skipped 1 previous similar message [ 1107.750228] Lustre: server umount lustre-MDT0000 complete [ 1109.482900] LustreError: 38986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1109.510891] LustreError: 38986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1111.717441] LustreError: 34979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831371 with bad export cookie 14609708138714590959 [ 1111.722594] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1111.725652] LustreError: 34979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1112.157965] Lustre: server umount lustre-MDT0001 complete [ 1123.281687] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1125.997189] Key type lgssc unregistered [ 1126.345231] LNet: 42865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.356197] LNetError: 42865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1126.394387] LNet: Removed LNI 192.168.204.102@tcp [ 1127.281201] Key type .llcrypt unregistered [ 1127.287658] Key type ._llcrypt unregistered [ 1146.024701] Key type ._llcrypt registered [ 1146.027733] Key type .llcrypt registered [ 1146.143900] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 1161.898564] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1162.948822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1162.971829] alg: No test for adler32 (adler32-zlib) [ 1164.126395] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1164.424328] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1166.087221] Key type lgssc registered [ 1167.343549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1208.938625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1208.997942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.447195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1210.494265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1210.635564] Lustre: lustre-MDT0000: new disk, initializing [ 1210.783105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.822128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1216.434778] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1226.747996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.899243] Lustre: 46021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1226.969459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1226.978222] Lustre: Skipped 1 previous similar message [ 1227.111260] Lustre: lustre-MDT0001: new disk, initializing [ 1227.266486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1227.329547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1227.343305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1232.389708] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1242.967642] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1248.795329] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1256.293583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1256.673420] Lustre: lustre-OST0000: new disk, initializing [ 1256.679198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1256.687818] Lustre: 47990:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1256.809069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.172240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1263.191593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1263.337625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1264.466959] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1275.019328] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1280.470543] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1280.727048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1286.177943] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1286.444951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1289.188456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1289.189649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.207381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1289.254508] Lustre: Skipped 1 previous similar message [ 1294.306857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1294.315019] Lustre: Skipped 2 previous similar messages [ 1294.729923] Lustre: server umount lustre-OST0000 complete [ 1304.034325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1304.047625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.066589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1306.138491] Lustre: server umount lustre-MDT0000 complete [ 1310.376321] LustreError: 46816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831570 with bad export cookie 8174855198299050281 [ 1310.377117] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1310.391083] LustreError: 46816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1310.697640] Lustre: server umount lustre-MDT0001 complete [ 1321.078881] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 21:39:39 (1788831579) [ 1328.305863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.755085] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.776879] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1328.840937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.351765] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1334.241952] LustreError: 50452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.365248] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.058522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.408395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1347.947278] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1357.718483] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1364.756192] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1373.102056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1373.640568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.272188] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1390.858330] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1395.862430] LustreError: 52979:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1397.160282] Lustre: Modifying parameter general.timeout=1 in log params [ 1399.204086] Lustre: Modifying parameter general.timeout=3 in log params [ 1399.215082] Lustre: Skipped 1 previous similar message [ 1401.738676] Lustre: Modifying parameter general.timeout=5 in log params [ 1401.744030] Lustre: Skipped 1 previous similar message [ 1406.566460] Lustre: Modifying parameter general.timeout=9 in log params [ 1406.573832] Lustre: Skipped 3 previous similar messages [ 1414.698474] Lustre: Modifying parameter general.timeout=15 in log params [ 1414.711233] Lustre: Skipped 5 previous similar messages [ 1430.931301] Lustre: Modifying parameter general.timeout=29 in log params [ 1430.937981] Lustre: Skipped 13 previous similar messages [ 1463.408672] Lustre: Modifying parameter general.timeout=58 in log params [ 1463.414673] Lustre: Skipped 28 previous similar messages [ 1528.247348] Lustre: Modifying parameter general.timeout=120 in log params [ 1528.265978] Lustre: Skipped 61 previous similar messages [ 1550.984608] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 21:43:29 (1788831809) [ 1554.245211] Lustre: 59937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1556.445446] Lustre: 60036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1558.474935] Lustre: 60134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1562.982015] Lustre: 60344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1562.994309] Lustre: 60344:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1596.134618] Lustre: 61818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1596.141832] Lustre: 61818:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1596.176115] LustreError: 61818:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1600.983357] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 21:44:19 (1788831859) [ 1616.869495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.875131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1616.888753] Lustre: Skipped 3 previous similar messages [ 1616.898658] Lustre: Skipped 3 previous similar messages [ 1621.336864] Lustre: server umount lustre-MDT0000 complete [ 1621.987548] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1622.009782] LustreError: 50451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1626.084909] LustreError: 50432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831885 with bad export cookie 8174855198299051163 [ 1626.097636] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1626.099533] LustreError: 50432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1626.555197] Lustre: server umount lustre-MDT0001 complete [ 1640.779559] Lustre: server umount lustre-OST0000 complete [ 1672.827912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1673.198652] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1673.229616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1673.238365] Lustre: Skipped 32 previous similar messages [ 1673.525320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1678.404835] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1687.999513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1688.108742] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1688.135817] Lustre: 64350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1693.383301] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1702.441995] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1709.277760] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1718.038144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1718.299496] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1718.604482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.611765] Lustre: Skipped 1 previous similar message [ 1725.647595] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1734.894524] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1736.649977] Lustre: 66918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1746.915908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1746.927270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.958055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1750.393361] Lustre: server umount lustre-OST0000 complete [ 1757.669032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.677392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.702741] Lustre: Skipped 2 previous similar messages [ 1757.721992] Lustre: Skipped 3 previous similar messages [ 1761.584269] Lustre: server umount lustre-MDT0000 complete [ 1762.797496] LustreError: 64358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.839385] LustreError: 64358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1766.485839] LustreError: 64344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832026 with bad export cookie 8174855198299054320 [ 1766.489931] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1766.496654] LustreError: 64344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1767.176222] Lustre: server umount lustre-MDT0001 complete [ 1778.669363] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1782.792706] Key type lgssc unregistered [ 1783.205927] LNet: 68122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.216311] LNetError: 68122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.265496] LNet: Removed LNI 192.168.204.102@tcp [ 1784.364573] Key type .llcrypt unregistered [ 1784.367309] Key type ._llcrypt unregistered [ 1811.067169] Key type ._llcrypt registered [ 1811.071352] Key type .llcrypt registered [ 1811.230064] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1813.090984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1813.157333] alg: No test for adler32 (adler32-zlib) [ 1814.223312] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1814.490249] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1816.191462] Key type lgssc registered [ 1817.416031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1825.986645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1826.011271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.539446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.736187] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1838.051505] LustreError: 69534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.063115] LustreError: 69534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1842.379632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.057676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1847.713531] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1855.898443] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1861.709326] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1869.792332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1870.524991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1878.331804] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1889.244932] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1896.972733] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1908.706660] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.710448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.727553] Lustre: Skipped 2 previous similar messages [ 1908.740363] Lustre: Skipped 2 previous similar messages [ 1913.826785] LustreError: 69528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.844670] LustreError: 69528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1913.961840] Lustre: server umount lustre-MDT0000 complete [ 1917.989972] LustreError: 69513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832177 with bad export cookie 9485693649547372920 [ 1917.997397] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1918.002734] LustreError: 69513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1918.491525] Lustre: server umount lustre-MDT0001 complete [ 1933.538934] Lustre: server umount lustre-OST0000 complete [ 1961.513926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.808885] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1961.854496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1962.156189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.209180] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1976.209539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.316277] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1976.345984] Lustre: 74409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1976.986051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1982.264482] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1991.092918] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1996.874383] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2005.376142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2005.765449] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2006.444784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.593668] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2023.967334] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2026.569970] Lustre: 76985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2028.909858] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2038.240476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2038.249193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.252425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2038.276531] Lustre: Skipped 1 previous similar message [ 2041.493746] Lustre: server umount lustre-OST0000 complete [ 2048.490813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2048.491768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2048.496514] LustreError: Skipped 1 previous similar message [ 2048.499563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2048.508965] Lustre: Skipped 1 previous similar message [ 2048.526525] Lustre: Skipped 2 previous similar messages [ 2052.450288] Lustre: server umount lustre-MDT0000 complete [ 2053.606089] LustreError: 74416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.624367] LustreError: 74416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2056.734479] LustreError: 76289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832316 with bad export cookie 9485693649547373774 [ 2056.746130] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.755685] LustreError: 76289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2057.253634] Lustre: server umount lustre-MDT0001 complete [ 2068.399756] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2072.776793] Key type lgssc unregistered [ 2073.385922] LNet: 78199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.402621] LNetError: 78199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.435404] LNet: Removed LNI 192.168.204.102@tcp [ 2074.664272] Key type .llcrypt unregistered [ 2074.666942] Key type ._llcrypt unregistered [ 2100.483759] Key type ._llcrypt registered [ 2100.488455] Key type .llcrypt registered [ 2100.632150] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2101.533217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2101.578802] alg: No test for adler32 (adler32-zlib) [ 2102.750665] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 2102.934090] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2104.583793] Key type lgssc registered [ 2106.195360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2116.749567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2116.772107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.349972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.336584] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2127.848425] LustreError: 79610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.883759] LustreError: 79610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2132.968458] LustreError: 79609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2133.398320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.261689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2138.666072] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2148.134562] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2155.251662] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2162.691667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.331883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2170.105682] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2178.263932] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2185.848171] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2186.885098] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2188.800784] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2188.817614] Lustre: Skipped 1 previous similar message [ 2191.778616] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2191.785911] Lustre: Skipped 2 previous similar messages [ 2195.966601] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2195.980986] Lustre: Skipped 4 previous similar messages [ 2204.818748] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2204.827505] Lustre: Skipped 9 previous similar messages [ 2221.561535] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2221.564968] Lustre: Skipped 16 previous similar messages [ 2254.481790] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2254.486266] Lustre: Skipped 33 previous similar messages [ 2291.719930] Lustre: 87717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2294.953977] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:55:53 (1788832553) [ 2297.148815] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2299.227544] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:55:57 (1788832557) [ 2304.996661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.016434] Lustre: Skipped 1 previous similar message [ 2305.018795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2308.087931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2308.093420] Lustre: Skipped 2 previous similar messages [ 2313.193336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2313.207214] Lustre: Skipped 1 previous similar message [ 2318.304846] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2318.305451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2318.323930] Lustre: Skipped 1 previous similar message [ 2318.557969] Lustre: server umount lustre-OST0000 complete [ 2323.424829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2323.428739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.438913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2323.450443] Lustre: Skipped 1 previous similar message [ 2323.463854] Lustre: Skipped 1 previous similar message [ 2329.598713] Lustre: server umount lustre-MDT0000 complete [ 2333.686451] LustreError: 79606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.722514] LustreError: 79606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2334.740031] LustreError: 79589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832594 with bad export cookie 6370320743761789677 [ 2334.741111] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2334.760103] LustreError: 79589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2335.525120] Lustre: server umount lustre-MDT0001 complete [ 2351.278832] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2354.647146] Key type lgssc unregistered [ 2354.977562] LNet: 88967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.003648] LNetError: 88967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.019320] LNet: Removed LNI 192.168.204.102@tcp [ 2356.079631] Key type .llcrypt unregistered [ 2356.082278] Key type ._llcrypt unregistered [ 2361.833978] Key type ._llcrypt registered [ 2361.835988] Key type .llcrypt registered [ 2362.083304] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2368.044447] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules [ 2369.737664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2369.810810] alg: No test for adler32 (adler32-zlib) [ 2371.147509] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 2371.515296] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2373.207374] Key type lgssc registered [ 2374.996529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.614849] LustreError: 90214:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2396.054586] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2399.146352] LustreError: 90214:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2406.197639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2406.226846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.889385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.032992] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2418.152361] LustreError: 91216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.175448] LustreError: 91216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2423.282469] LustreError: 91215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.893894] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:58:02 (1788832682) [ 2428.388232] LustreError: 91216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.481187] LustreError: 91248:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2429.497951] LustreError: 91248:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2429.506419] LustreError: 91248:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2429.920403] Lustre: server umount lustre-MDT0000 complete [ 2446.492444] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2449.217500] Key type lgssc unregistered [ 2449.560674] LNet: 92740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.572050] LNetError: 92740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2449.590343] LNet: Removed LNI 192.168.204.102@tcp [ 2450.511554] Key type .llcrypt unregistered [ 2450.517909] Key type ._llcrypt unregistered [ 2471.602545] Key type ._llcrypt registered [ 2471.608577] Key type .llcrypt registered [ 2471.806933] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2474.002691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2474.052936] alg: No test for adler32 (adler32-zlib) [ 2475.273605] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 2475.587756] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2477.327299] Key type lgssc registered [ 2478.897949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2490.335812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2490.351870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.155387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.487738] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2501.606695] LustreError: 93958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.653869] LustreError: 93958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2506.721470] LustreError: 93959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.433933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.582219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2514.729275] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2523.825577] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2530.412722] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2538.282675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2539.063159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.133543] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2555.421870] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2616.122420] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:01:15 (1788832875) [ 2617.906264] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2619.797090] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:01:18 (1788832878) [ 2627.041709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2627.052192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.078768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2628.073513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.079219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.940523] Lustre: server umount lustre-MDT0000 complete [ 2633.198502] LustreError: 95254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.214810] LustreError: 95254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2633.662240] LustreError: 93939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832893 with bad export cookie 14141433636500118287 [ 2633.678705] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2633.680137] LustreError: 93939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2633.913778] Lustre: server umount lustre-MDT0001 complete [ 2648.433716] Lustre: server umount lustre-OST0000 complete [ 2660.663899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.057299] LustreError: 98288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.231838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.200222] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2666.467165] LustreError: 98287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.681814] LustreError: 98287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.713306] LustreError: 98287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2676.398508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.470182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2683.771368] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2693.602963] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2699.797936] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2715.469995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2715.826351] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2715.838225] LustreError: 98275:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2715.869330] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2715.885697] LustreError: 100372:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2715.899277] LustreError: 100372:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2715.903827] LustreError: 100372:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2715.993842] Lustre: server umount lustre-OST0000 complete [ 2715.996780] LustreError: 100372:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2725.568969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2725.739172] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2725.746403] LustreError: 98275:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2725.775776] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2725.800050] LustreError: 100837:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2725.807355] LustreError: 100837:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2725.813350] LustreError: 100837:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2725.915542] Lustre: server umount lustre-OST0000 complete [ 2725.917838] LustreError: 100837:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2736.464658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2736.661039] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2736.671246] Lustre: Found index 0 for lustre-OST0000, updating log [ 2736.680869] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2736.816752] Lustre: lustre-OST0000: new disk, initializing [ 2736.822635] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2736.829155] Lustre: 101372:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2737.154610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.355773] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2747.285599] Lustre: Failing over lustre-OST0000 [ 2747.514891] Lustre: server umount lustre-OST0000 complete [ 2753.504827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2753.507109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.512361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2753.512369] Lustre: Skipped 2 previous similar messages [ 2753.540314] Lustre: Skipped 2 previous similar messages [ 2757.684521] Lustre: server umount lustre-MDT0000 complete [ 2758.638437] LustreError: 98287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.655081] LustreError: 98287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2761.828045] LustreError: 101371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833021 with bad export cookie 14141433636500121024 [ 2761.828801] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2761.844219] LustreError: 101371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2762.167918] Lustre: server umount lustre-MDT0001 complete [ 2771.583876] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:03:50 (1788833030) [ 2790.090664] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2801.609313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.020936] LustreError: 103719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.154552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2807.160645] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2816.820422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.760295] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2826.410938] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2833.995148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2834.525929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2834.530646] Lustre: Skipped 1 previous similar message [ 2841.149614] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2850.695445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.779434] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2850.924880] Lustre: lustre-OST0001: new disk, initializing [ 2850.929201] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2850.941210] Lustre: 106035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2857.090554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2857.108630] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2857.256151] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2858.298611] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2867.203825] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2879.495899] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2884.584475] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2884.592623] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.609176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2884.614668] Lustre: Skipped 1 previous similar message [ 2892.418720] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.2@tcp (stopping) [ 2892.428701] Lustre: Skipped 4 previous similar messages [ 2898.403715] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2898.618677] Lustre: server umount lustre-MDT0001 complete [ 2902.680165] LustreError: 107022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2902.724837] LustreError: 107022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2910.226435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2910.398367] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2910.416689] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2910.554605] Lustre: 103708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2911.656323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2911.676083] Lustre: Skipped 1 previous similar message [ 2911.769371] LustreError: 107557:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2911.794450] LustreError: 107557:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2911.826696] LustreError: 103711:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2912.125206] LustreError: 107574:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.102@tcp: cfg command failed: rc = -17 [ 2912.144684] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2912.144684] [ 2916.489324] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2916.859497] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2928.514045] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:06:27 (1788833187) [ 2949.413301] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2969.061204] Lustre: 109655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2994.014826] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2997.786258] Lustre: 110791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3090.400751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3090.409512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.431871] Lustre: Skipped 2 previous similar messages [ 3090.448294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3090.457421] Lustre: Skipped 7 previous similar messages [ 3093.057821] Lustre: server umount lustre-MDT0000 complete [ 3096.033679] LustreError: 106769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.041776] LustreError: 106769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3097.445692] LustreError: 103699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833357 with bad export cookie 14141433636500122025 [ 3097.445849] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3097.465978] LustreError: 103699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3098.201672] Lustre: server umount lustre-MDT0001 complete [ 3113.108979] Lustre: server umount lustre-OST0000 complete [ 3117.023106] Lustre: 93269:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788833360/real 1788833360] req@ffff8a124f6a1f80 x1875727074311808/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788833376 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3117.053698] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.061885] Lustre: Skipped 3 previous similar messages [ 3117.286928] Lustre: server umount lustre-OST0001 complete [ 3123.404848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3133.675461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3142.482639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3153.439974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3173.403205] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3184.405702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.437943] Lustre: lustre-MDT0000: reset Object Index mappings [ 3185.183781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.923242] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3202.108465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3202.139857] Lustre: lustre-MDT0001: reset Object Index mappings [ 3208.427507] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3211.915434] Lustre: 115888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3219.959501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3227.534047] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3230.134686] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3230.589501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3240.597258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3246.588221] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3246.616642] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3248.314254] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3256.447849] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3260.702196] Lustre: 117763:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3313.869753] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:12:52 (1788833572) [ 3341.282542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3341.288861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3341.308344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3341.315256] Lustre: Skipped 3 previous similar messages [ 3346.607136] Lustre: server umount lustre-MDT0000 complete [ 3350.374574] LustreError: 117765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833610 with bad export cookie 14141433636500168918 [ 3350.380406] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3350.382941] LustreError: 117765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3354.081141] LustreError: 116156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.113417] LustreError: 116156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 3356.947320] Lustre: server umount lustre-MDT0001 complete [ 3375.071196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3375.266283] Lustre: server umount lustre-OST0000 complete [ 3385.799589] Lustre: server umount lustre-OST0001 complete [ 3392.932689] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 3404.172711] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3443.615862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.816270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3443.847199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3443.932888] Lustre: lustre-MDT0000: new disk, initializing [ 3444.032113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3444.037471] Lustre: Skipped 3 previous similar messages [ 3444.056147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3448.944116] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3456.477859] Lustre: server umount lustre-MDT0000 complete [ 3466.610941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3466.848634] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3466.860141] Lustre: 122042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3466.873631] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3471.283569] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3476.688836] Lustre: server umount lustre-MDT0000 complete [ 3485.465621] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:15:44 (1788833744) [ 3487.197687] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3489.423561] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:15:48 (1788833748) [ 3507.486717] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3518.128811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.612809] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3532.879407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.045170] Lustre: 124092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3533.086321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3533.088899] Lustre: Skipped 1 previous similar message [ 3533.149403] Lustre: lustre-MDT0001: new disk, initializing [ 3533.267722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3533.275436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3538.185939] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3543.425503] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3551.383041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3551.714853] Lustre: lustre-OST0000: new disk, initializing [ 3551.722553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3551.729969] Lustre: 125669:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3552.950312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3552.965982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3553.122285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3558.721073] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3569.658714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3569.836511] Lustre: lustre-OST0001: new disk, initializing [ 3569.844191] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3569.849604] Lustre: 126540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3571.378781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3571.390326] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3571.515645] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3576.207702] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3585.108770] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3588.384828] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4023.947319] Lustre: 132967:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4023.955214] Lustre: 132967:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4023.961618] Lustre: 132967:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4023.969742] Lustre: 132967:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4023.976697] Lustre: 132967:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4023.987771] Lustre: 132967:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4330.267235] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:29:48 (1788834588) [ 4347.360589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4347.374596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.400669] Lustre: Skipped 5 previous similar messages [ 4347.416167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4347.429637] Lustre: Skipped 8 previous similar messages [ 4351.543285] Lustre: server umount lustre-MDT0000 complete [ 4352.998936] LustreError: 127369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.027930] LustreError: 127369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4355.612960] LustreError: 132996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834615 with bad export cookie 14141433636500238554 [ 4355.617172] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4355.629649] LustreError: 132996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4358.114916] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.124995] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4358.130290] Lustre: Skipped 4 previous similar messages [ 4358.139713] Lustre: Skipped 4 previous similar messages [ 4362.288364] Lustre: server umount lustre-MDT0001 complete [ 4381.151172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4381.281724] Lustre: server umount lustre-OST0000 complete [ 4391.398251] Lustre: server umount lustre-OST0001 complete [ 4400.319972] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 4414.392381] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 4456.668149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.995651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4457.033817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4457.159969] Lustre: lustre-MDT0000: new disk, initializing [ 4457.241646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4457.246342] Lustre: Skipped 5 previous similar messages [ 4457.260257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4461.706219] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4473.549596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4473.699202] Lustre: 136027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4473.803872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4473.807330] Lustre: Skipped 1 previous similar message [ 4473.889803] Lustre: lustre-MDT0001: new disk, initializing [ 4474.079989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4474.105358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4479.741779] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4489.572537] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4495.787937] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4502.640180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4502.945665] Lustre: lustre-OST0000: new disk, initializing [ 4502.949539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4502.957289] Lustre: 137996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4505.050867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4505.056742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4505.239987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4509.040598] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4517.982071] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4529.922572] Lustre: lustre-MDD0000: changelog on [ 4532.706302] Lustre: lustre-MDD0001: changelog on [ 6647.748352] Lustre: lustre-MDD0001: changelog off [ 6649.774964] Lustre: 137641:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6650.747411] Lustre: lustre-MDD0000: changelog off [ 6660.579220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6660.592496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6660.594544] LustreError: Skipped 1 previous similar message [ 6660.595927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6660.595934] Lustre: Skipped 2 previous similar messages [ 6660.629814] Lustre: Skipped 1 previous similar message [ 6664.280703] Lustre: server umount lustre-OST0000 complete [ 6670.819242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6670.822980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6670.862637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6670.884625] Lustre: Skipped 2 previous similar messages [ 6675.938130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6675.961718] Lustre: Skipped 1 previous similar message [ 6681.062585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6681.077091] Lustre: Skipped 1 previous similar message [ 6684.127669] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6684.833626] Lustre: server umount lustre-MDT0000 complete [ 6686.184076] LustreError: 136037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6686.217807] LustreError: 136037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6690.477083] LustreError: 136019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836950 with bad export cookie 14141433636501059668 [ 6690.478801] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6690.497399] LustreError: 136019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6691.402367] Lustre: server umount lustre-MDT0001 complete [ 6705.295398] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 6708.571839] Key type lgssc unregistered [ 6708.990556] LNet: 147198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6709.002121] LNetError: 147198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6710.054057] LNet: Removed LNI 192.168.204.102@tcp [ 6711.191208] Key type .llcrypt unregistered [ 6711.194764] Key type ._llcrypt unregistered [ 6735.472607] Key type ._llcrypt registered [ 6735.477806] Key type .llcrypt registered [ 6735.622665] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 6752.523330] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 6753.742873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6753.791878] alg: No test for adler32 (adler32-zlib) [ 6754.987197] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 6755.424606] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 6757.175221] Key type lgssc registered [ 6759.049454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6801.489982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6801.535690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6802.846932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6802.888222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6803.021495] Lustre: lustre-MDT0000: new disk, initializing [ 6803.137303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6803.172078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6808.426745] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6820.316293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6820.531237] Lustre: 150399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6820.727524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6820.737962] Lustre: Skipped 1 previous similar message [ 6820.852591] Lustre: lustre-MDT0001: new disk, initializing [ 6820.958554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6821.026897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6821.043428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6826.259531] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6835.969887] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6843.359822] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6853.420676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6854.020305] Lustre: lustre-OST0000: new disk, initializing [ 6854.036628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6854.061608] Lustre: 152369:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6854.271565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6860.900428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6860.921399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6861.122642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6864.045671] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6876.003971] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6883.473086] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_module obdecho/obdecho [ 6885.983168] LustreError: 153372:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6885.996523] LustreError: 153372:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6888.245944] LustreError: 153469:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6907.368572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6907.382819] Lustre: Skipped 1 previous similar message [ 6907.387935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6907.394905] Lustre: Skipped 1 previous similar message [ 6909.824145] Lustre: server umount lustre-MDT0000 complete [ 6915.502602] LustreError: 150389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837175 with bad export cookie 4134411145414650116 [ 6915.506269] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6915.537261] LustreError: 150389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6915.881854] Lustre: server umount lustre-MDT0001 complete [ 6932.425078] Lustre: server umount lustre-OST0000 complete [ 6955.550349] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 6968.229821] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7008.883345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.182435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7009.270627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7009.364371] Lustre: lustre-MDT0000: new disk, initializing [ 7009.452210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7009.488786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7015.506845] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7026.261690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.395891] Lustre: 157187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7026.468449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7026.474120] Lustre: Skipped 1 previous similar message [ 7026.568922] Lustre: lustre-MDT0001: new disk, initializing [ 7026.680872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7026.735277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7026.746501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7031.640215] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7041.999391] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7047.852972] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7055.275256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7055.571597] Lustre: lustre-OST0000: new disk, initializing [ 7055.575496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7055.584791] Lustre: 159155:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7055.700754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.045378] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7063.692186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7063.701497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7063.840943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7071.971815] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7078.333122] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7078.694666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7084.550093] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7084.902578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7089.634526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7089.648879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7089.681077] Lustre: Skipped 1 previous similar message [ 7089.694930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7089.703646] Lustre: Skipped 2 previous similar messages [ 7093.557364] Lustre: server umount lustre-OST0000 complete [ 7098.336934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7098.344918] LustreError: Skipped 1 previous similar message [ 7098.351469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.367970] Lustre: Skipped 1 previous similar message [ 7098.378258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7104.605382] Lustre: server umount lustre-MDT0000 complete [ 7104.993221] LustreError: 157194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7105.033440] LustreError: 157194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7109.978110] LustreError: 157178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837369 with bad export cookie 4134411145414651075 [ 7109.984149] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7109.989172] LustreError: 157178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7110.016379] LustreError: 148617:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 7110.633543] Lustre: server umount lustre-MDT0001 complete [ 7115.697701] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:16:14 (1788837374) [ 7117.740734] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7119.917885] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:16:18 (1788837378) [ 7129.926713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.601554] LustreError: 161572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7130.697784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7135.712829] LustreError: 161573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7137.025653] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7140.846605] LustreError: 161572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.954934] LustreError: 161573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7148.407537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.617881] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7167.292310] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7173.900221] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7182.090087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7182.580068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7182.590247] Lustre: Skipped 1 previous similar message [ 7189.547322] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7198.117095] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7214.233233] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.2@tcp (stopping) [ 7214.253019] Lustre: Skipped 2 previous similar messages [ 7215.590840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7215.599612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7215.612874] Lustre: Skipped 1 previous similar message [ 7216.622903] LustreError: 161573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7216.848251] Lustre: server umount lustre-MDT0000 complete [ 7222.011406] LustreError: 161553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837481 with bad export cookie 4134411145414651957 [ 7222.035322] LustreError: 161553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7222.037574] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7224.489990] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.2@tcp (stopping) [ 7224.495081] Lustre: Skipped 1 previous similar message [ 7226.857571] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7226.880616] Lustre: Skipped 2 previous similar messages [ 7228.783747] Lustre: server umount lustre-MDT0001 complete [ 7236.823709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7250.580479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7250.813892] LustreError: 165440:0:(llog.c:1655:llog_backup()) MGC192.168.204.102@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7250.819774] Lustre: 165440:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.102@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7251.436857] LustreError: 165445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.460257] LustreError: 165445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7284.868341] LustreError: 165446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7284.899531] LustreError: 165446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 7311.839990] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7311.848558] Lustre: 165440:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.102@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7350.415386] LustreError: 165446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7350.435083] LustreError: 165446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 7352.844850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7357.937742] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7358.519215] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7369.767576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.526912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7398.391826] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7398.943160] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7408.648684] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7410.282959] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7416.583552] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7418.395496] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7439.336201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7439.357387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7439.369239] Lustre: Skipped 2 previous similar messages [ 7439.369485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7439.402887] Lustre: Skipped 4 previous similar messages [ 7440.193138] Lustre: server umount lustre-MDT0000 complete [ 7444.290628] LustreError: 161552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837703 with bad export cookie 4134411145414658446 [ 7444.295676] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7444.304506] LustreError: 161552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7444.842543] Lustre: server umount lustre-MDT0001 complete [ 7450.627287] Lustre: server umount lustre-OST0000 complete [ 7478.974987] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 7491.143434] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7534.864660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.152927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7535.202137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7535.370287] Lustre: lustre-MDT0000: new disk, initializing [ 7535.542455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7535.550377] Lustre: Skipped 1 previous similar message [ 7535.578084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7540.346684] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7551.871379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7552.065586] Lustre: 170552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7552.496655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7557.859548] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7568.570619] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7575.902684] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7583.700172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7584.087775] Lustre: lustre-OST0000: new disk, initializing [ 7584.090911] Lustre: Skipped 1 previous similar message [ 7584.098859] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7584.103405] Lustre: Skipped 2 previous similar messages [ 7584.121156] Lustre: 172518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7591.593386] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7593.019886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7593.026717] Lustre: Skipped 1 previous similar message [ 7593.036479] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7593.176648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7601.519132] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7606.810919] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7607.043863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7612.242146] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7612.565174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7619.052195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7619.055061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.066859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7619.070895] Lustre: Skipped 2 previous similar messages [ 7619.081832] Lustre: Skipped 2 previous similar messages [ 7629.279161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7629.501814] Lustre: server umount lustre-OST0000 complete [ 7640.373063] Lustre: server umount lustre-MDT0000 complete [ 7644.650498] LustreError: 170563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7644.678942] LustreError: 170563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 7645.086732] LustreError: 174062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837904 with bad export cookie 4134411145414675239 [ 7645.095318] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7645.110361] LustreError: 174062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7645.936680] Lustre: server umount lustre-MDT0001 complete [ 7656.039431] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:25:15 (1788837915) [ 7663.332610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.379321] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7678.023741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7683.500984] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7692.519754] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7699.369259] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7707.404424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7715.852511] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7725.658371] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7738.155438] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:26:36 (1788837996) [ 7743.976078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7743.991951] LustreError: Skipped 2 previous similar messages [ 7743.999226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7744.018672] Lustre: Skipped 2 previous similar messages [ 7744.026593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7744.033269] Lustre: Skipped 9 previous similar messages [ 7748.834377] Lustre: server umount lustre-OST0000 complete [ 7759.590496] Lustre: server umount lustre-MDT0000 complete [ 7764.157320] LustreError: 174914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838023 with bad export cookie 4134411145414676114 [ 7764.164360] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7764.170665] LustreError: 174914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7764.613623] Lustre: server umount lustre-MDT0001 complete [ 7778.981282] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 7782.254383] Key type lgssc unregistered [ 7782.539983] LNet: 178664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7782.556764] LNetError: 178664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7782.582515] LNet: Removed LNI 192.168.204.102@tcp [ 7783.611388] Key type .llcrypt unregistered [ 7783.614114] Key type ._llcrypt unregistered [ 7807.185919] Key type ._llcrypt registered [ 7807.187803] Key type .llcrypt registered [ 7807.344301] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7808.880701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7809.089247] alg: No test for adler32 (adler32-zlib) [ 7810.296763] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 7810.532215] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 7812.231198] Key type lgssc registered [ 7814.251421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7824.189795] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7824.206827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7839.775447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7844.895604] LustreError: 179904:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 7850.016284] LustreError: 179904:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7850.039984] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7850.044232] LustreError: MGC192.168.204.102@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7850.051065] LustreError: 179904:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7850.057440] LustreError: 179904:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7850.075451] LustreError: 179904:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7850.174160] Lustre: server umount lustre-OST0000 complete [ 7850.180154] LustreError: 179904:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7859.881493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7861.285463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7866.521845] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7870.444614] LustreError: 180402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7870.478894] LustreError: 180402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7875.554669] LustreError: 180403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7876.697522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7877.268243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7882.528766] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7892.735164] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7899.728777] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7907.138737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7907.330386] LustreError: 182222:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7907.346337] Lustre: 182222:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.102@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7907.514929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7913.954840] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7922.143916] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7924.372204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7924.390514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7924.409535] Lustre: Skipped 2 previous similar messages [ 7924.429556] Lustre: Skipped 2 previous similar messages [ 7929.832084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7929.852407] Lustre: Skipped 2 previous similar messages [ 7930.707846] Lustre: server umount lustre-MDT0000 complete [ 7934.808504] LustreError: 180383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838194 with bad export cookie 721890031144816052 [ 7934.830404] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7934.958698] LustreError: 180398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7934.967552] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.005557] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7935.157201] Lustre: server umount lustre-MDT0001 complete [ 7945.315972] Lustre: server umount lustre-OST0000 complete [ 7955.875296] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 7968.025749] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8009.395541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8009.585594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8009.609368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8009.683060] Lustre: lustre-MDT0000: new disk, initializing [ 8009.781921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8009.801213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8014.458620] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8026.421191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8026.543230] Lustre: 185746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8026.600352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8026.604841] Lustre: Skipped 1 previous similar message [ 8026.693713] Lustre: lustre-MDT0001: new disk, initializing [ 8026.806900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8026.884643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8026.895417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8031.936751] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8040.670683] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8047.212698] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8054.940482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8055.275160] Lustre: lustre-OST0000: new disk, initializing [ 8055.279366] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8055.286251] Lustre: 187714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8055.377085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8061.989809] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8064.090442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8064.106718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8064.261136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8071.808325] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8077.203847] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8077.435364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8081.984410] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8082.194880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8084.960153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8084.964550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.979983] LustreError: Skipped 1 previous similar message [ 8084.980983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8085.005902] Lustre: Skipped 1 previous similar message [ 8090.401736] Lustre: server umount lustre-OST0000 complete [ 8098.276699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8098.279545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8098.303115] Lustre: Skipped 1 previous similar message [ 8098.307125] Lustre: Skipped 3 previous similar messages [ 8100.682116] Lustre: server umount lustre-MDT0000 complete [ 8103.395761] LustreError: 187378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.421632] LustreError: 187378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8105.874213] LustreError: 185738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838365 with bad export cookie 721890031144816759 [ 8105.885569] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8105.899250] LustreError: 185738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8106.355852] Lustre: server umount lustre-MDT0001 complete [ 8117.828456] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:32:56 (1788838376) [ 8137.733276] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 8141.121203] Key type lgssc unregistered [ 8141.506777] LNet: 190657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8141.512815] LNetError: 190657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8141.532858] LNet: Removed LNI 192.168.204.102@tcp [ 8142.754112] Key type .llcrypt unregistered [ 8142.756354] Key type ._llcrypt unregistered [ 8165.474489] Key type ._llcrypt registered [ 8165.479546] Key type .llcrypt registered [ 8165.651540] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8166.945134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8166.998690] alg: No test for adler32 (adler32-zlib) [ 8168.328944] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8168.685568] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 8170.463234] Key type lgssc registered [ 8172.102286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8181.943804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8181.990311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8183.614716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.008783] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8194.023095] LustreError: 191874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8194.051027] LustreError: 191874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8198.915306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.283996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8204.417303] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8212.869114] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8219.872338] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8227.673599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8227.945867] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8227.951597] LustreError: 193739:0:(llog.c:1696:llog_backup()) MGC192.168.204.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8227.959855] Lustre: 193739:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.102@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8228.074713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8235.138901] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8242.815591] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8248.801656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.805645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8248.828851] Lustre: Skipped 2 previous similar messages [ 8248.855059] Lustre: Skipped 2 previous similar messages [ 8251.947270] Lustre: server umount lustre-MDT0000 complete [ 8253.932249] LustreError: 191874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.973751] LustreError: 191874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8256.690200] LustreError: 191854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838516 with bad export cookie 3604793385316005225 [ 8256.695677] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8256.700293] LustreError: 191854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8257.123221] Lustre: server umount lustre-MDT0001 complete [ 8270.806687] Lustre: server umount lustre-OST0000 complete [ 8281.802712] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 8292.346970] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8332.408401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8332.603497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8332.638667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8332.765373] Lustre: lustre-MDT0000: new disk, initializing [ 8332.867214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8332.895506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8337.580210] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8349.538438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8349.687269] Lustre: 197262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8349.743861] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8349.746199] Lustre: Skipped 1 previous similar message [ 8349.749613] LustreError: 198046:0:(llog.c:1696:llog_backup()) MGC192.168.204.102@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8349.756019] LustreError: 198046:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8349.761403] Lustre: 198046:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.102@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8349.786447] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8349.789510] Lustre: Skipped 1 previous similar message [ 8349.903380] Lustre: lustre-MDT0001: new disk, initializing [ 8350.078121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8350.145164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8350.159597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8355.593941] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8367.333396] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8373.262720] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8380.986267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8381.249212] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8381.251584] Lustre: Skipped 1 previous similar message [ 8381.254234] LustreError: 199209:0:(llog.c:1696:llog_backup()) MGC192.168.204.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8381.264719] LustreError: 199209:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8381.270789] Lustre: 199209:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.102@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8381.280150] Lustre: 199209:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8381.301656] Lustre: lustre-OST0000: new disk, initializing [ 8381.305921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8381.313836] Lustre: 199232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8381.415978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8388.187560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8388.205847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8388.398865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8388.725532] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8400.163531] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8405.746238] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8405.985602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8413.240813] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8413.572789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8419.300107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8419.304542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8419.318451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8419.338290] Lustre: Skipped 1 previous similar message [ 8421.829905] Lustre: server umount lustre-OST0000 complete [ 8426.978061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8426.988898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8427.002018] Lustre: Skipped 1 previous similar message [ 8427.005423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8429.538520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.541116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8429.568085] Lustre: Skipped 1 previous similar message [ 8434.658790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8434.675382] Lustre: Skipped 1 previous similar message [ 8440.287567] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8440.487911] Lustre: server umount lustre-MDT0000 complete [ 8444.389180] LustreError: 197255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838704 with bad export cookie 3604793385316005946 [ 8444.389945] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8444.403552] LustreError: 197255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8444.767319] Lustre: server umount lustre-MDT0001 complete [ 8453.479519] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:38:32 (1788838712) [ 8478.071614] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8489.918905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.438412] LustreError: 202383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8490.528744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8495.584721] LustreError: 202384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8495.825785] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8500.710380] LustreError: 202383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8504.700979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.497923] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8512.597123] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8519.049791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8526.274445] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8535.693162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8535.933063] Lustre: lustre-OST0001: new disk, initializing [ 8535.938872] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8535.961330] Lustre: 204690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8536.122637] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8536.131121] Lustre: Skipped 2 previous similar messages [ 8543.128604] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8543.380397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8543.404991] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8543.643904] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8554.146500] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8561.500672] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8563.682130] Lustre: DEBUG MARKER: ADD OST3 [ 8569.132846] loop: module loaded [ 8569.258554] loop0: detected capacity change from 0 to 204800000 [ 8569.522950] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8569.539157] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8569.553454] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8575.288744] loop0: detected capacity change from 0 to 204800000 [ 8576.230864] Lustre: DEBUG MARKER: STOP OST3 [ 8580.946658] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8581.182260] Lustre: lustre-OST0003: new disk, initializing [ 8581.186969] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8581.194248] Lustre: 206684:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8582.960916] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8583.087799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8583.092530] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8583.143248] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8584.992443] LustreError: 206863:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8585.795867] Lustre: Failing over lustre-OST0003 [ 8586.109202] Lustre: server umount lustre-OST0003 complete [ 8587.948718] LustreError: 203870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8588.285197] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8588.285695] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8588.326868] Lustre: Skipped 1 previous similar message [ 8589.233471] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8598.695939] Lustre: DEBUG MARKER: START OST3 again [ 8605.957145] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8606.275868] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8606.281976] Lustre: Skipped 1 previous similar message [ 8606.297171] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8607.523632] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8608.367709] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8608.382081] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8613.198158] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8635.855124] Lustre: Failing over lustre-OST0003 [ 8636.195977] Lustre: server umount lustre-OST0003 complete [ 8636.916345] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8636.933246] LustreError: 203869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8636.945723] LustreError: 203869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8657.382422] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8657.385910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8657.402575] Lustre: Skipped 3 previous similar messages [ 8657.412775] Lustre: Skipped 5 previous similar messages [ 8661.882060] Lustre: server umount lustre-MDT0000 complete [ 8665.765498] LustreError: 204689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838925 with bad export cookie 3604793385316006828 [ 8665.768370] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8665.776388] LustreError: 204689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8670.625441] LustreError: 202379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8670.650870] LustreError: 202379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8672.228820] Lustre: server umount lustre-MDT0001 complete [ 8677.063795] Lustre: server umount lustre-OST0000 complete [ 8681.819816] Lustre: server umount lustre-OST0001 complete [ 8689.937385] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 8699.325672] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8742.042932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8742.264486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8742.308559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8742.373294] Lustre: lustre-MDT0000: new disk, initializing [ 8742.448095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8742.486634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8747.797686] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8760.170408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8760.366782] Lustre: 212151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8760.473798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8760.482632] Lustre: Skipped 1 previous similar message [ 8760.632555] Lustre: lustre-MDT0001: new disk, initializing [ 8760.824853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8760.833824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8766.124690] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8777.655343] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8783.629919] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8790.982076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8791.208373] Lustre: 214122:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8793.003050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8793.014497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8793.169740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8798.158530] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8809.757840] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8815.617402] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8815.871672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8822.463524] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8822.813548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8828.897236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8828.928699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8828.947213] Lustre: Skipped 3 previous similar messages [ 8828.954075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8828.965837] Lustre: Skipped 5 previous similar messages [ 8839.138345] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8839.513934] Lustre: server umount lustre-OST0000 complete [ 8850.855092] Lustre: server umount lustre-MDT0000 complete [ 8854.507504] LustreError: 212161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8854.539567] LustreError: 212161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8855.717661] LustreError: 212946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839115 with bad export cookie 3604793385316009124 [ 8855.719232] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8855.729359] LustreError: 212946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8856.107378] Lustre: server umount lustre-MDT0001 complete [ 8860.292517] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:45:19 (1788839119) [ 8879.025833] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 8890.147614] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8927.740854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8927.987423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8928.012692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8928.017279] Lustre: Skipped 1 previous similar message [ 8928.082218] Lustre: lustre-MDT0000: new disk, initializing [ 8928.084634] Lustre: Skipped 1 previous similar message [ 8928.169478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8933.247071] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8946.720321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.797793] Lustre: 218735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8947.178176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8951.727083] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8964.153706] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8973.546496] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8983.493633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8983.803088] Lustre: 220707:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8985.156842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8985.385757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8992.286330] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9003.369629] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9009.602546] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9009.931163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9016.657911] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9016.803810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9021.432068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9021.433769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9021.446065] LustreError: Skipped 2 previous similar messages [ 9021.459795] Lustre: Skipped 3 previous similar messages [ 9021.461832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9021.461842] Lustre: Skipped 8 previous similar messages [ 9025.429789] Lustre: server umount lustre-OST0000 complete [ 9036.691362] Lustre: server umount lustre-MDT0000 complete [ 9036.784208] LustreError: 219621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9040.524443] LustreError: 218726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839300 with bad export cookie 3604793385316010006 [ 9040.537377] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9040.538833] LustreError: 218726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9041.067221] Lustre: server umount lustre-MDT0001 complete [ 9050.771042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9051.230071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9051.234607] Lustre: Skipped 5 previous similar messages [ 9055.690056] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9064.901730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9070.601464] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9079.145506] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9085.306365] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9093.133769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9101.236549] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9109.520951] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9120.229936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9120.250359] Lustre: Skipped 4 previous similar messages [ 9126.191781] Lustre: server umount lustre-OST0000 complete [ 9144.287230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9144.794602] Lustre: server umount lustre-MDT0000 complete [ 9150.170707] LustreError: 224769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839409 with bad export cookie 3604793385316010888 [ 9150.184649] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9150.192599] LustreError: 224769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9151.045291] Lustre: server umount lustre-MDT0001 complete [ 9250.622911] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 9253.828655] Key type lgssc unregistered [ 9254.236636] LNet: 226591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9254.256218] LNetError: 226591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9254.288259] LNet: Removed LNI 192.168.204.102@tcp [ 9255.295618] Key type .llcrypt unregistered [ 9255.300721] Key type ._llcrypt unregistered [ 9286.323123] Key type ._llcrypt registered [ 9286.326346] Key type .llcrypt registered [ 9286.481359] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 9305.285431] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9307.244959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9307.275757] alg: No test for adler32 (adler32-zlib) [ 9308.498031] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 9308.897739] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 9310.687875] Key type lgssc registered [ 9312.071749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9356.305467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9356.374916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9357.719469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9357.768696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9357.887067] Lustre: lustre-MDT0000: new disk, initializing [ 9358.001409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9358.052961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9363.979861] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9375.169922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9375.310068] Lustre: 229944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9375.391547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9375.397893] Lustre: Skipped 1 previous similar message [ 9375.499158] Lustre: lustre-MDT0001: new disk, initializing [ 9375.616494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9375.673210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9375.690223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9381.309147] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9391.726184] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9398.573863] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9407.165694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9407.511668] Lustre: lustre-OST0000: new disk, initializing [ 9407.517730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9407.531799] Lustre: 231912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9407.678704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9415.119211] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9416.326823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9416.346932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9416.536492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9425.821974] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9430.599975] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9430.809919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9436.185072] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9436.408274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9442.292499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9442.301314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9442.305272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9442.311728] Lustre: Skipped 1 previous similar message [ 9442.355690] LustreError: Skipped 1 previous similar message [ 9444.782732] Lustre: server umount lustre-OST0000 complete [ 9452.521149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9452.526620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9452.532257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9452.532267] Lustre: Skipped 1 previous similar message [ 9452.582101] Lustre: Skipped 1 previous similar message [ 9457.635455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9457.646140] Lustre: Skipped 2 previous similar messages [ 9462.753606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9462.761956] Lustre: Skipped 1 previous similar message [ 9464.287141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9464.711182] Lustre: server umount lustre-MDT0000 complete [ 9468.776059] LustreError: 230740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839728 with bad export cookie 16073803525482107372 [ 9468.779348] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9468.792443] LustreError: 230740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9469.206310] Lustre: server umount lustre-MDT0001 complete [ 9491.179247] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9503.936746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9504.432270] LustreError: 234848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9504.449897] LustreError: 234848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9504.509040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9509.142170] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9509.856499] LustreError: 234849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9514.987993] LustreError: 234848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9519.761572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9520.318543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9526.469100] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9529.880977] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9537.568382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9537.970667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.707448] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9554.931957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9555.136364] Lustre: lustre-OST0001: new disk, initializing [ 9555.139860] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9555.146468] Lustre: 237157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9555.237832] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9556.101088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9556.114461] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9556.250899] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9562.726620] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9575.016150] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9582.315076] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9599.771533] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 23:57:38 (1788839858) [ 9602.531774] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9602.538263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9602.567697] Lustre: Skipped 2 previous similar messages [ 9602.575462] Lustre: Skipped 1 previous similar message [ 9607.138056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9607.149974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9608.603222] Lustre: server umount lustre-MDT0000 complete [ 9612.771770] LustreError: 236929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9612.808939] LustreError: 236929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9613.260839] LustreError: 234831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839872 with bad export cookie 16073803525482108254 [ 9613.277816] LustreError: 234831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9613.280857] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9613.688518] Lustre: server umount lustre-MDT0001 complete [ 9628.554837] Lustre: server umount lustre-OST0000 complete [ 9643.327882] Lustre: server umount lustre-OST0001 complete [ 9650.368322] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 9660.781596] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9699.768066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9699.935429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9699.958530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9700.016770] Lustre: lustre-MDT0000: new disk, initializing [ 9700.090767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9700.106594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9705.320270] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9717.870914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9718.042451] Lustre: 241102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9718.135568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9718.147675] Lustre: Skipped 1 previous similar message [ 9718.232454] Lustre: lustre-MDT0001: new disk, initializing [ 9718.460236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9718.469880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9723.378755] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9733.689974] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9741.175416] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9748.620825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9748.863869] Lustre: lustre-OST0000: new disk, initializing [ 9748.867732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9748.873869] Lustre: 243074:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9750.477089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9750.490848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9750.604609] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9755.367787] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9765.797431] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9770.853240] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9771.056658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9776.329253] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9776.616064] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9781.240012] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9781.253830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9781.272978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9781.285813] Lustre: Skipped 6 previous similar messages [ 9784.922749] Lustre: server umount lustre-OST0000 complete [ 9789.920835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9789.933383] LustreError: Skipped 1 previous similar message [ 9789.946670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9789.961067] Lustre: Skipped 1 previous similar message [ 9795.585646] Lustre: server umount lustre-MDT0000 complete [ 9796.576819] LustreError: 241111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9796.603232] LustreError: 241111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9799.588276] LustreError: 241096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840059 with bad export cookie 16073803525482110389 [ 9799.600449] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9799.609656] LustreError: 241096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9799.877955] Lustre: server umount lustre-MDT0001 complete [ 9809.063591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.533684] LustreError: 245304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.646225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9809.659404] Lustre: Skipped 2 previous similar messages [ 9814.571991] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9823.977692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9830.014870] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9839.210967] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9845.281982] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9853.104095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9860.678973] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9868.452341] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10017.765583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10017.775666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10017.783821] Lustre: Skipped 2 previous similar messages [10017.791047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10017.804351] Lustre: Skipped 4 previous similar messages [10020.569951] Lustre: server umount lustre-OST0000 complete [10039.263563] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10039.680480] Lustre: server umount lustre-MDT0000 complete [10043.362091] LustreError: 245305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10043.396464] LustreError: 245305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10044.211400] LustreError: 245286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840303 with bad export cookie 16073803525482111271 [10044.212402] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10044.224584] LustreError: 245286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10044.577743] Lustre: server umount lustre-MDT0001 complete [10139.929760] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [10144.008950] Key type lgssc unregistered [10144.439239] LNet: 248911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10144.450934] LNetError: 248911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10144.467536] LNet: Removed LNI 192.168.204.102@tcp [10145.741369] Key type .llcrypt unregistered [10145.745871] Key type ._llcrypt unregistered [10179.312050] Key type ._llcrypt registered [10179.314655] Key type .llcrypt registered [10179.439899] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [10200.796333] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [10202.115664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10202.181236] alg: No test for adler32 (adler32-zlib) [10203.377519] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [10203.730551] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [10205.455217] Key type lgssc registered [10206.937298] Lustre: Echo OBD driver; http://www.lustre.org/ [10261.392606] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [10273.678787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10273.716350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10275.026546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10275.064181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10275.153915] Lustre: lustre-MDT0000: new disk, initializing [10275.238406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10275.251848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10281.010956] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10294.460074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10294.646729] Lustre: 252980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10294.713661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10294.727080] Lustre: Skipped 1 previous similar message [10294.867869] Lustre: lustre-MDT0001: new disk, initializing [10295.003776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10295.050839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10295.062382] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10301.268519] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10307.375862] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10315.321186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10315.725245] Lustre: lustre-OST0000: new disk, initializing [10315.737745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10315.753353] Lustre: 254613:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10315.867640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10320.493852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10320.505559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10320.648125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10322.726727] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10335.229626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10335.512440] Lustre: lustre-OST0001: new disk, initializing [10335.516623] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10335.529351] Lustre: 255486:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10335.655332] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10341.524730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10341.547230] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10341.704748] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10343.688455] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10356.680821] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10364.327472] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10377.184635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10377.200708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10377.233460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10377.700688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10377.721472] Lustre: Skipped 2 previous similar messages [10382.824409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10382.831309] Lustre: Skipped 4 previous similar messages [10383.138436] LustreError: 256566:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff8a136123c800) refcount nonzero (1) after lock cleanup; forcing cleanup. [10385.280452] Lustre: server umount lustre-MDT0000 complete [10390.377669] LustreError: 252974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840650 with bad export cookie 2242082760104126256 [10390.383837] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10390.402225] LustreError: 252974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10390.826553] Lustre: server umount lustre-MDT0001 complete [10406.097309] Lustre: server umount lustre-OST0000 complete [10408.993138] Lustre: 250485:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840652/real 1788840652] req@ffff8a13473af480 x1875735177410432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788840668 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10409.055109] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10410.016658] Lustre: server umount lustre-OST0001 complete [10424.833714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10435.857300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10444.943567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10455.705748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10477.469928] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [10491.746040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10491.788794] Lustre: lustre-MDT0000: reset Object Index mappings [10492.249207] LustreError: 260497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10492.291373] LustreError: 260497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10492.383188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10497.504746] LustreError: 260498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10498.024708] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10501.617027] LustreError: 260497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10506.726485] LustreError: 260498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10509.317116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10509.356569] Lustre: lustre-MDT0001: reset Object Index mappings [10509.943638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10515.440901] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10518.662355] Lustre: 261643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10527.279626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10527.836332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10535.031230] LustreError: 261999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10535.827945] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10537.098615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10546.411892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10551.832794] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10554.628120] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10565.172129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10569.863866] Lustre: 263515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10576.353691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10576.358140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10576.371237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10582.503343] LustreError: 262223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10582.524114] LustreError: 262223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10582.674309] Lustre: server umount lustre-MDT0000 complete [10587.198183] LustreError: 260479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840846 with bad export cookie 2242082760104133053 [10587.215288] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10587.220872] LustreError: 260479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10587.680349] Lustre: server umount lustre-MDT0001 complete [10603.323366] Lustre: server umount lustre-OST0000 complete [10604.259777] Lustre: 250484:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840847/real 1788840847] req@ffff8a13473fb100 x1875735177465472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788840863 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10604.295801] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10604.325782] Lustre: Skipped 3 previous similar messages [10607.455271] Lustre: server umount lustre-OST0001 complete [10631.428988] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:14:49 (1788840889) [10651.403442] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [10666.723119] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [10708.846524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10709.259608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10709.301548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10709.409271] Lustre: lustre-MDT0000: new disk, initializing [10709.550811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10709.562366] Lustre: Skipped 1 previous similar message [10709.617754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10715.284435] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10727.805483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10727.984070] Lustre: 267799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10728.044319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10728.047808] Lustre: Skipped 1 previous similar message [10728.146600] Lustre: lustre-MDT0001: new disk, initializing [10728.297933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10728.323992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10733.478749] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10745.479819] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10752.261673] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10759.877764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10760.219769] Lustre: lustre-OST0000: new disk, initializing [10760.226075] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10760.243436] Lustre: 269766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10761.355955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10761.372187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10761.531574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10766.230649] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10777.076058] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10783.193241] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10783.527478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10788.852677] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10789.106035] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10792.423662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10792.427456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10792.434876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10792.434887] Lustre: Skipped 4 previous similar messages [10792.438391] Lustre: Skipped 1 previous similar message [10797.551449] LustreError: 270661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10797.572149] LustreError: 270661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10797.693244] Lustre: server umount lustre-OST0000 complete [10802.663522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10802.672053] Lustre: Skipped 2 previous similar messages [10808.302512] Lustre: server umount lustre-MDT0000 complete [10813.044438] LustreError: 267791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841072 with bad export cookie 2242082760104135146 [10813.055387] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10813.058545] LustreError: 267791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10813.565967] Lustre: server umount lustre-MDT0001 complete [10838.203958] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [10851.348741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.873482] LustreError: 272700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.899506] LustreError: 272700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10851.996336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10852.012217] Lustre: Skipped 2 previous similar messages [10857.056918] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10866.160763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10871.359979] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10874.859674] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10882.927455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10891.490148] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10901.291704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10901.483822] Lustre: lustre-OST0001: new disk, initializing [10901.486866] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10901.495052] Lustre: 275010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10902.677455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10902.687428] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10902.781778] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10909.405707] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10920.059707] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10924.117746] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10929.542716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10929.621190] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10938.337910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10938.353207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10938.363277] Lustre: Skipped 2 previous similar messages [10938.374449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10938.384166] Lustre: Skipped 2 previous similar messages [10951.135383] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10951.465505] Lustre: server umount lustre-MDT0000 complete [10954.215539] LustreError: 272696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10954.260560] LustreError: 272696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10955.374944] LustreError: 272682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841215 with bad export cookie 2242082760104136028 [10955.378110] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10955.390627] LustreError: 272682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10955.796138] Lustre: server umount lustre-MDT0001 complete [10971.347386] Lustre: server umount lustre-OST0000 complete [10975.200936] Lustre: 250486:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788841218/real 1788841218] req@ffff8a124a202a00 x1875735177577728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788841234 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10975.228343] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10975.247937] Lustre: Skipped 3 previous similar messages [10975.856169] Lustre: server umount lustre-OST0001 complete [10997.221798] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [11011.971621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11012.522656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11012.532685] Lustre: Skipped 3 previous similar messages [11018.319562] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11030.849621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11037.403564] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11041.508552] Lustre: 279211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11049.211223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11056.578925] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11057.800950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11066.471486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11072.024400] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11073.863485] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11082.925906] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11086.698882] Lustre: 281086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11092.254637] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11092.287779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11102.673599] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:22:41 (1788841361) [11118.562227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11118.578101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11118.600965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11118.607892] Lustre: Skipped 11 previous similar messages [11124.835289] Lustre: server umount lustre-MDT0000 complete [11128.289917] LustreError: 278071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11128.322984] LustreError: 278071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [11128.708526] LustreError: 281088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841388 with bad export cookie 2242082760104142573 [11128.716083] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11128.721302] LustreError: 281088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11143.135476] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11143.353496] Lustre: server umount lustre-MDT0001 complete [11153.789298] Lustre: server umount lustre-OST0000 complete [11158.458250] Lustre: server umount lustre-OST0001 complete [11165.699640] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [11180.270267] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [11228.297652] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [11240.404819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11240.687377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11240.723422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11240.856601] Lustre: lustre-MDT0000: new disk, initializing [11241.009357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11247.547424] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11261.251657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11261.381475] Lustre: 285341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11261.787787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11267.223871] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11279.244887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11279.499091] Lustre: 286976:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11279.603336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11279.607960] Lustre: Skipped 5 previous similar messages [11281.246193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11285.298659] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11296.370493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11296.573940] Lustre: 287847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11297.986568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11297.997542] Lustre: Skipped 1 previous similar message [11298.179352] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11303.699501] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11313.706699] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11317.474561] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11317.479755] Lustre: Skipped 1 previous similar message [11331.490919] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:26:30 (1788841590) [11333.603169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11333.608569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11333.627094] Lustre: Skipped 5 previous similar messages [11333.632595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11333.637596] Lustre: Skipped 8 previous similar messages [11335.838733] Lustre: server umount lustre-MDT0000 complete [11339.528168] LustreError: 287846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841599 with bad export cookie 2242082760104150217 [11339.535438] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11339.543622] LustreError: 287846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11339.969412] Lustre: server umount lustre-MDT0001 complete [11353.723953] Lustre: server umount lustre-OST0000 complete [11367.898350] Lustre: server umount lustre-OST0001 complete [11376.373618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11380.864777] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11387.362480] LustreError: 290096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11387.379530] LustreError: 290096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11388.807429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11393.070458] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11400.456898] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11406.645578] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11413.691370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11421.790559] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11432.152864] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11447.542429] loop1: detected capacity change from 0 to 204800000 [11447.670334] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11447.678800] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11447.692697] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11453.135547] loop1: detected capacity change from 0 to 204800000 [11459.776336] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11459.904338] LustreError: 290085:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs30004: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11459.917884] LustreError: 290085:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs30004 [11459.925696] LustreError: 290085:0:(mgs_handler.c:517:mgs_target_reg()) fs30004-MDT0001: Error checking target index: rc = -13 [11459.933260] LustreError: fs30004-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11459.951716] LustreError: 293816:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11459.960089] LustreError: 293816:0:(tgt_mount.c:2132:server_put_super()) no obd fs30004-MDT0001 [11459.969090] LustreError: 293816:0:(tgt_mount.c:116:server_deregister_mount()) fs30004-MDT0001 not registered [11460.020226] Lustre: server umount fs30004-MDT0001 complete [11460.023439] LustreError: 293816:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11469.815363] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11469.907951] Lustre: Setting parameter fs30004-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs30004-MDT0001 [11469.952493] Lustre: srv-fs30004-MDT0001: No data found on store. Initialize space. [11469.958269] Lustre: Skipped 4 previous similar messages [11470.089856] Lustre: fs30004-MDT0001: new disk, initializing [11470.101455] Lustre: Skipped 3 previous similar messages [11470.393030] Lustre: srv-fs30004-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11470.403276] Lustre: 294283:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30004-MDT0001-mdtlov: can't create new llog: rc = -115 [11470.427171] LustreError: 294283:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30004-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11475.821796] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11483.609094] Lustre: server umount fs30004-MDT0001 complete [11493.357580] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11497.109880] loop1: detected capacity change from 0 to 204800000 [11503.792716] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11504.429689] Lustre: 295851:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30004-MDT0001-mdtlov: can't create new llog: rc = -115 [11504.442284] LustreError: 295851:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30004-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11509.327679] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11514.564086] Lustre: server umount fs30004-MDT0001 complete [11527.648540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11527.664618] LustreError: Skipped 1 previous similar message [11527.682283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11527.688308] Lustre: Skipped 4 previous similar messages [11531.963986] Lustre: server umount lustre-MDT0000 complete [11543.370723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11543.554645] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11549.165873] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11549.547400] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11558.107774] loop1: detected capacity change from 0 to 204800000 [11564.427468] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11564.801713] Lustre: 298299:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs30004-MDT0001-mdtlov: can't create new llog: rc = -115 [11564.809108] LustreError: 298299:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs30004-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11569.583747] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11575.398189] Lustre: server umount fs30004-MDT0001 complete [11598.822199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11598.839708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11598.858382] Lustre: Skipped 6 previous similar messages [11604.832659] Lustre: server umount lustre-OST0000 complete [11615.154336] Lustre: server umount lustre-MDT0000 complete [11619.586891] LustreError: 291949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841879 with bad export cookie 2242082760104152898 [11619.601455] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11619.606901] LustreError: 291949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11620.130220] Lustre: server umount lustre-MDT0001 complete [11633.218482] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [11636.543618] Key type lgssc unregistered [11636.796577] LNet: 300643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11636.802489] LNetError: 300643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11636.823316] LNet: Removed LNI 192.168.204.102@tcp [11637.790268] Key type .llcrypt unregistered [11637.796968] Key type ._llcrypt unregistered [11658.795351] Key type ._llcrypt registered [11658.800585] Key type .llcrypt registered [11658.884838] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [11659.836443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11659.914494] alg: No test for adler32 (adler32-zlib) [11660.982564] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [11661.219401] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [11662.944638] Key type lgssc registered [11664.674543] Lustre: Echo OBD driver; http://www.lustre.org/ [11675.122700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11675.159979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11676.728944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11682.779500] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11686.896821] LustreError: 301914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.931984] LustreError: 301914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11687.915933] LustreError: 301913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11693.026921] LustreError: 301914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.181887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11694.603750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11699.432276] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11707.365476] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11713.631459] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11721.428368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11721.828651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11722.871310] LustreError: 303742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11728.361662] LustreError: 303744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11729.027887] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11738.171519] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11738.596229] LustreError: 303742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11738.623176] LustreError: 303742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11754.988690] LustreError: 303744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11754.997593] LustreError: 303744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11759.570044] loop1: detected capacity change from 0 to 204800000 [11759.818902] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11759.833727] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11759.844743] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11766.226812] loop1: detected capacity change from 0 to 204800000 [11771.984391] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11772.182198] Lustre: fs7437-OST0000: new disk, initializing [11772.186393] Lustre: srv-fs7437-OST0000: No data found on store. Initialize space. [11772.194322] Lustre: 305650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11772.309276] Lustre: fs7437-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11779.417610] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11787.400391] LustreError: 303744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11787.427396] LustreError: 303744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11788.207583] Lustre: server umount fs7437-OST0000 complete [11801.080236] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11802.821283] loop1: detected capacity change from 0 to 204800000 [11809.242806] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11809.698308] Lustre: fs7437-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11816.662200] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11821.557471] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11826.111816] loop2: detected capacity change from 0 to 204800000 [11826.457489] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11826.487903] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11826.504364] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11832.925164] loop2: detected capacity change from 0 to 204800000 [11839.726787] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11839.907040] LustreError: fs7437-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11839.933474] LustreError: fs7437-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11839.943245] LustreError: 308692:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11839.946359] LustreError: 308692:0:(tgt_mount.c:2132:server_put_super()) no obd fs7437-OST0001 [11839.964894] LustreError: 308692:0:(tgt_mount.c:116:server_deregister_mount()) fs7437-OST0001 not registered [11839.994795] Lustre: server umount fs7437-OST0001 complete [11839.999105] LustreError: 308692:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11851.813779] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11852.214349] Lustre: fs7437-OST0001: new disk, initializing [11852.229892] Lustre: srv-fs7437-OST0001: No data found on store. Initialize space. [11852.248876] Lustre: 309243:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11852.271877] LustreError: 303799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11852.300092] LustreError: 303799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [11852.505662] Lustre: fs7437-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11860.016602] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11869.299904] Lustre: server umount fs7437-OST0000 complete [11879.861774] Lustre: server umount fs7437-OST0001 complete [11893.421330] loop1: detected capacity change from 0 to 204800000 [11900.056381] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11900.386471] Lustre: fs7437-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11907.116612] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11915.001382] loop2: detected capacity change from 0 to 204800000 [11921.265642] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11928.492343] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11937.941639] Lustre: server umount fs7437-OST0000 complete [11947.495469] Lustre: server umount fs7437-OST0001 complete [11954.677092] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11963.873101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11963.878559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11963.886825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11964.902313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11964.909612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11966.180764] Lustre: server umount lustre-OST0000 complete [11971.055525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11971.072423] LustreError: Skipped 1 previous similar message [11971.080403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11971.094668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11971.104380] Lustre: Skipped 1 previous similar message [11975.136611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11975.141250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11975.184709] Lustre: Skipped 1 previous similar message [11980.256663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11980.272707] Lustre: Skipped 1 previous similar message [11985.379974] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11985.755714] Lustre: server umount lustre-MDT0000 complete [11990.447579] LustreError: 306395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842250 with bad export cookie 6358615768144654171 [11990.466066] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11990.469965] LustreError: 306395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11991.108852] Lustre: server umount lustre-MDT0001 complete [12001.898752] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [12005.518486] Key type lgssc unregistered [12005.760734] LNet: 315168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12005.769070] LNetError: 315168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12005.781154] LNet: Removed LNI 192.168.204.102@tcp [12006.952386] Key type .llcrypt unregistered [12006.954149] Key type ._llcrypt unregistered [12040.992364] Key type ._llcrypt registered [12040.995652] Key type .llcrypt registered [12041.120191] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [12058.326803] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [12059.790892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12059.929821] alg: No test for adler32 (adler32-zlib) [12060.994263] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [12061.207873] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [12062.943186] Key type lgssc registered [12064.559648] Lustre: Echo OBD driver; http://www.lustre.org/ [12110.414563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12110.464038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12111.673350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12111.719278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12111.798319] Lustre: lustre-MDT0000: new disk, initializing [12111.958112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12111.981062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12117.338992] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12128.643173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12128.771311] Lustre: 319026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12128.829618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12128.832589] Lustre: Skipped 1 previous similar message [12128.931985] Lustre: lustre-MDT0001: new disk, initializing [12129.056816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12129.112871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12129.134218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12133.402433] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12143.062930] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12148.254961] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12154.677443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12155.019600] Lustre: lustre-OST0000: new disk, initializing [12155.025361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12155.031721] Lustre: 320992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12155.130236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12161.619803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12161.632878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12161.727621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12161.885318] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12171.506667] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12173.468792] Lustre: Mounted lustre-client - version 2.17.57_104_ga33798f [12174.572937] Lustre: Unmounted lustre-client [12187.618161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12187.620329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12187.633798] LustreError: Skipped 1 previous similar message [12187.638334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12187.642516] Lustre: Skipped 1 previous similar message [12189.165773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12189.177490] Lustre: Skipped 2 previous similar messages [12190.580708] Lustre: server umount lustre-OST0000 complete [12195.809114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12195.819108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12195.836645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12199.392518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12199.393707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12199.415919] Lustre: Skipped 1 previous similar message [12204.513361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12204.522814] Lustre: Skipped 1 previous similar message [12209.119483] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12209.297680] Lustre: server umount lustre-MDT0000 complete [12213.139484] LustreError: 319018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842472 with bad export cookie 15521238547809115434 [12213.145794] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12213.153420] LustreError: 319018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12213.550284] Lustre: server umount lustre-MDT0001 complete [12225.706762] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [12229.312241] Key type lgssc unregistered [12230.688666] LNet: 323153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12230.710402] LNetError: 323153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12305.509956] LNet: Removed LNI 192.168.204.102@tcp [12307.082433] Key type .llcrypt unregistered [12307.089877] Key type ._llcrypt unregistered [12326.695437] Key type ._llcrypt registered [12326.699329] Key type .llcrypt registered [12326.828324] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [12345.815903] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [12347.194561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12347.303256] alg: No test for adler32 (adler32-zlib) [12348.703459] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [12349.109133] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [12350.887181] Key type lgssc registered [12352.856328] Lustre: Echo OBD driver; http://www.lustre.org/ [12388.982497] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:44:07 (1788842647) [12396.158684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12396.178098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12397.406292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12397.439608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12397.520407] Lustre: lustre-MDT0000: new disk, initializing [12397.592493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12397.605768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12401.491706] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12410.221706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12410.375168] Lustre: 326375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12410.426067] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12410.430370] Lustre: Skipped 1 previous similar message [12410.502249] Lustre: lustre-MDT0001: new disk, initializing [12410.584487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12410.626777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12410.646016] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12414.995810] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12424.009470] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12429.961628] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12437.380517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12437.820685] Lustre: lustre-OST0000: new disk, initializing [12437.831438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12437.852611] Lustre: 328344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12437.998528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12444.800065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12444.817463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12445.000165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12445.380290] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12456.299722] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12475.876985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12475.878232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12475.879568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12475.923854] Lustre: Skipped 1 previous similar message [12479.715892] Lustre: server umount lustre-OST0000 complete [12486.119339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12486.122816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12486.130989] Lustre: Skipped 1 previous similar message [12486.141085] Lustre: Skipped 2 previous similar messages [12491.030978] Lustre: server umount lustre-MDT0000 complete [12495.344300] LustreError: 326367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842754 with bad export cookie 7043417424453304175 [12495.356938] LustreError: 326367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12495.362383] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12495.813348] Lustre: server umount lustre-MDT0001 complete [12507.852313] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [12510.715582] Key type lgssc unregistered [12511.043979] LNet: 330250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12511.055463] LNetError: 330250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12511.068920] LNet: Removed LNI 192.168.204.102@tcp [12511.910592] Key type .llcrypt unregistered [12511.915575] Key type ._llcrypt unregistered [12534.071845] Key type ._llcrypt registered [12534.073509] Key type .llcrypt registered [12534.176715] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [12535.076459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12535.223656] alg: No test for adler32 (adler32-zlib) [12536.353022] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [12536.535892] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [12538.175190] Key type lgssc registered [12539.683939] Lustre: Echo OBD driver; http://www.lustre.org/ [12549.242228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12549.300445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12551.109624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12555.739048] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12561.381063] LustreError: 331524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12561.408644] LustreError: 331524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12564.900239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12565.343221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12569.785864] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12577.624138] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12584.121715] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12591.530447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12592.081630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12598.564595] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12606.413587] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12621.618498] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12623.848626] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12629.728406] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.2@tcp (stopping) [12629.990103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12630.032630] Lustre: Skipped 1 previous similar message [12634.756136] LustreError: 331518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12635.052818] Lustre: server umount lustre-MDT0000 complete [12639.873493] LustreError: 331518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12639.927661] LustreError: 331518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12641.319622] LustreError: 331505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842900 with bad export cookie 7103188401372461469 [12641.336307] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12641.339074] LustreError: 331505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12645.002934] LustreError: 331520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12645.034026] LustreError: 331520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12645.056870] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.2@tcp (stopping) [12645.073616] Lustre: Skipped 3 previous similar messages [12645.346539] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12645.371694] Lustre: Skipped 1 previous similar message [12646.371238] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12646.387767] Lustre: Skipped 1 previous similar message [12647.741833] Lustre: server umount lustre-MDT0001 complete [12652.936911] Lustre: server umount lustre-OST0000 complete [12664.397040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12670.379960] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12673.653359] Lustre: 335709:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.102@tcp [12676.260257] Lustre: server umount MGS complete [12685.178310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12685.617124] LustreError: 336280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12685.640717] LustreError: 336280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12685.687789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12690.187588] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12696.034485] LustreError: 336281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12696.053606] LustreError: 336281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12698.642866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12699.014307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12703.476140] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12712.642409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12713.190711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12719.580480] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12729.238479] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12730.834348] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12737.835984] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12739.743416] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12746.212151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12746.224173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12746.233340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12746.252157] Lustre: Skipped 1 previous similar message [12746.276404] Lustre: Skipped 1 previous similar message [12751.340768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12751.353535] Lustre: Skipped 1 previous similar message [12751.849662] Lustre: server umount lustre-OST0000 complete [12756.449271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12761.569722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12761.583681] Lustre: Skipped 3 previous similar messages [12762.358026] Lustre: server umount lustre-MDT0000 complete [12766.689195] LustreError: 337008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12766.709063] LustreError: 337008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12766.987580] LustreError: 336261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843026 with bad export cookie 7103188401372462316 [12766.997732] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12767.381764] Lustre: server umount lustre-MDT0001 complete [12779.279277] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [12783.317394] Key type lgssc unregistered [12783.655523] LNet: 339716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12783.677092] LNetError: 339716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12783.705845] LNet: Removed LNI 192.168.204.102@tcp [12785.157599] Key type .llcrypt unregistered [12785.162578] Key type ._llcrypt unregistered [12807.787296] Key type ._llcrypt registered [12807.789908] Key type .llcrypt registered [12807.933073] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [12826.414674] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [12827.103591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12827.114334] alg: No test for adler32 (adler32-zlib) [12828.435470] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [12829.129212] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [12830.999240] Key type lgssc registered [12832.165722] Lustre: Echo OBD driver; http://www.lustre.org/ [12872.780090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12872.806399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12874.074164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12874.093958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12874.181321] Lustre: lustre-MDT0000: new disk, initializing [12874.255708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12874.275909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12878.839827] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12890.651258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12890.756729] Lustre: 342877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12890.810730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12890.814592] Lustre: Skipped 1 previous similar message [12890.910131] Lustre: lustre-MDT0001: new disk, initializing [12891.056701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12891.140904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12891.155472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12896.106374] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12906.858193] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12913.752485] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12921.442993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12921.705638] Lustre: lustre-OST0000: new disk, initializing [12921.709900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12921.721705] Lustre: 344843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12921.836703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12929.287095] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12931.628144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12931.639793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12931.782715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12940.795950] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12947.391522] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12947.811792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12953.862299] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12954.159446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12957.666212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12957.677559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12957.695340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12957.705029] Lustre: Skipped 1 previous similar message [12962.370378] Lustre: server umount lustre-OST0000 complete [12967.915461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12967.916449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12967.925121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12967.945620] Lustre: Skipped 2 previous similar messages [12967.987051] LustreError: Skipped 1 previous similar message [12973.028154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12973.043240] Lustre: Skipped 2 previous similar messages [12973.966137] Lustre: server umount lustre-MDT0000 complete [12978.520808] LustreError: 342868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843238 with bad export cookie 16788168470030253886 [12978.530030] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12978.541524] LustreError: 342868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12979.011866] Lustre: server umount lustre-MDT0001 complete [12989.929573] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 00:54:08 (1788843248) [13010.095549] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [13028.172103] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [13072.722347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13073.058941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13073.092679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13073.251888] Lustre: lustre-MDT0000: new disk, initializing [13073.323295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13073.337552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13079.198631] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13092.402807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13092.594719] Lustre: 349627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13092.652440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13092.656953] Lustre: Skipped 1 previous similar message [13092.796312] Lustre: lustre-MDT0001: new disk, initializing [13092.961784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13093.030248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13093.060989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13097.667513] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13107.699480] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13113.873796] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13122.253785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13122.742566] Lustre: lustre-OST0000: new disk, initializing [13122.759636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13122.771429] Lustre: 351600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13122.897540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13130.360050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13130.391447] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13130.544097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13131.252906] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13141.448929] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13176.806658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13176.808765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13176.810043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13176.812101] LustreError: Skipped 1 previous similar message [13176.833803] Lustre: Skipped 1 previous similar message [13179.658850] Lustre: server umount lustre-OST0000 complete [13184.992846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13185.008726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13185.037350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13185.049650] Lustre: Skipped 1 previous similar message [13189.561447] Lustre: server umount lustre-MDT0000 complete [13192.160154] LustreError: 350793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13192.168306] LustreError: 350793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13193.378468] LustreError: 349620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843453 with bad export cookie 16788168470030254768 [13193.386322] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13193.398031] LustreError: 349620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13193.835209] Lustre: server umount lustre-MDT0001 complete [13206.720553] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13209.776401] Key type lgssc unregistered [13210.153622] LNet: 353773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13210.167129] LNetError: 353773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13210.194319] LNet: Removed LNI 192.168.204.102@tcp [13211.023839] Key type .llcrypt unregistered [13211.026770] Key type ._llcrypt unregistered [13311.713054] Key type ._llcrypt registered [13311.718462] Key type .llcrypt registered [13311.930417] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13312.401125] Key type .llcrypt unregistered [13312.408056] Key type ._llcrypt unregistered [13398.834865] Key type ._llcrypt registered [13398.837310] Key type .llcrypt registered [13398.929765] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13399.219452] Key type .llcrypt unregistered [13399.222671] Key type ._llcrypt unregistered [13432.787525] Key type ._llcrypt registered [13432.790766] Key type .llcrypt registered [13432.928720] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13433.360299] Key type .llcrypt unregistered [13433.365719] Key type ._llcrypt unregistered [13469.908637] Key type ._llcrypt registered [13469.911350] Key type .llcrypt registered [13470.062477] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13470.400595] Key type .llcrypt unregistered [13470.403442] Key type ._llcrypt unregistered [13555.958410] Key type ._llcrypt registered [13555.960906] Key type .llcrypt registered [13556.133256] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13556.580619] Key type .llcrypt unregistered [13556.582259] Key type ._llcrypt unregistered [13591.916658] Key type ._llcrypt registered [13591.920411] Key type .llcrypt registered [13592.251709] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13592.722447] Key type .llcrypt unregistered [13592.725749] Key type ._llcrypt unregistered [13627.284950] Key type ._llcrypt registered [13627.287444] Key type .llcrypt registered [13627.406485] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13627.678420] Key type .llcrypt unregistered [13627.683631] Key type ._llcrypt unregistered [13665.372922] Key type ._llcrypt registered [13665.383313] Key type .llcrypt registered [13665.547481] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13665.838517] Key type .llcrypt unregistered [13665.845277] Key type ._llcrypt unregistered [13701.420679] Key type ._llcrypt registered [13701.425522] Key type .llcrypt registered [13701.551297] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13701.944053] Key type .llcrypt unregistered [13701.949272] Key type ._llcrypt unregistered [13736.336663] Key type ._llcrypt registered [13736.346756] Key type .llcrypt registered [13736.517795] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13736.793627] Key type .llcrypt unregistered [13736.798580] Key type ._llcrypt unregistered [13808.610721] Key type ._llcrypt registered [13808.613553] Key type .llcrypt registered [13808.739575] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13809.062476] Key type .llcrypt unregistered [13809.066045] Key type ._llcrypt unregistered [13848.528673] Key type ._llcrypt registered [13848.535069] Key type .llcrypt registered [13848.722683] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13849.133209] Key type .llcrypt unregistered [13849.138838] Key type ._llcrypt unregistered [13884.686553] Key type ._llcrypt registered [13884.690429] Key type .llcrypt registered [13884.823709] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13885.144413] Key type .llcrypt unregistered [13885.155090] Key type ._llcrypt unregistered [13948.270875] Key type ._llcrypt registered [13948.274265] Key type .llcrypt registered [13948.417290] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [13965.807937] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [13967.014652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13967.066081] alg: No test for adler32 (adler32-zlib) [13968.145496] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [13968.456215] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [13970.193517] Key type lgssc registered [13971.241915] Lustre: Echo OBD driver; http://www.lustre.org/ [14014.964173] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14014.993184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14016.316410] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14016.355466] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14016.435362] Lustre: tst250fs-MDT0000: new disk, initializing [14016.542391] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14016.565380] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14021.634213] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [14033.963074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14034.164801] Lustre: 374077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14034.266767] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14034.271257] Lustre: Skipped 1 previous similar message [14034.406740] Lustre: tst250fs-MDT0001: new disk, initializing [14034.610148] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14034.663570] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14034.674623] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14040.900160] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [14047.146261] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14055.228184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14055.451205] Lustre: tst250fs-OST0000: new disk, initializing [14055.462370] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14055.470034] Lustre: 375713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14055.593573] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14062.608545] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [14063.211662] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14063.226731] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14063.404816] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14073.446124] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14073.706277] Lustre: tst250fs-OST0001: new disk, initializing [14073.714783] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14073.722893] Lustre: 376585:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14073.889772] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14081.778690] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [14083.760291] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14083.783435] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14083.918374] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14088.670072] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14091.154166] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14091.161723] Lustre: Skipped 1 previous similar message [14114.801405] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14114.807935] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14114.814075] Lustre: Skipped 2 previous similar messages [14114.833233] Lustre: Skipped 3 previous similar messages [14118.493754] Lustre: server umount tst250fs-MDT0000 complete [14122.721974] LustreError: 374072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844382 with bad export cookie 8403938407121930870 [14122.726074] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14122.731437] LustreError: 374072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14123.273738] Lustre: server umount tst250fs-MDT0001 complete [14138.448516] Lustre: server umount tst250fs-OST0000 complete [14141.279140] Lustre: 372301:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844384/real 1788844384] req@ffff8a1247a54000 x1875739125290624/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14141.302105] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14141.310975] Lustre: Skipped 1 previous similar message [14142.432514] Lustre: 372302:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844386/real 1788844386] req@ffff8a124b138380 x1875739125290880/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844402 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14142.818749] Lustre: server umount tst250fs-OST0001 complete [14166.670421] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [14178.092817] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [14220.752164] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:14:38 (1788844478) [14223.030265] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14244.051481] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [14260.547727] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [14301.786732] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14044 sec ========= 01:15:59 (1788844559) [14303.808592] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:16:02 (1788844562) === [14308.101647] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:16:06 (1788844566) === [14353.993987] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [14357.141685] Key type lgssc unregistered [14357.516993] LNet: 385748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14357.523107] LNetError: 385748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14357.545992] LNet: Removed LNI 192.168.204.102@tcp [14358.522164] Key type .llcrypt unregistered [14358.523639] Key type ._llcrypt unregistered